Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1484066 Details for
Bug 1629889
Device removal leaves garbage on the device returning "ok", breaking further attempts to "add" device back
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
Heketi server logs
heketi_server_logs_add_remove_device_problems (text/plain), 2.93 MB, created by
Valerii Ponomarov
on 2018-09-17 14:58:42 UTC
(
hide
)
Description:
Heketi server logs
Filename:
MIME Type:
Creator:
Valerii Ponomarov
Created:
2018-09-17 14:58:42 UTC
Size:
2.93 MB
patch
obsolete
>[heketi] INFO 2018/09/17 00:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 00:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 00:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 00:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 00:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 00:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 00:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 00:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 00:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 00:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 00:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 00:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:22:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:22:00 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:22:00 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:22:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:22:00 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:22:00 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:24:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:24:00 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:24:00 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 01:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 01:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 01:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 01:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 01:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 01:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 01:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 01:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 01:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 01:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 01:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 02:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 02:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 02:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 02:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 02:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 02:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 02:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 02:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 02:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 02:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 02:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 03:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 03:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 03:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 03:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 03:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 03:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 03:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 03:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 03:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 03:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 03:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 04:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 04:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 04:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 04:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 04:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 04:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 04:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 04:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 04:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 04:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 04:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 05:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 05:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 05:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 05:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 05:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 05:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 05:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 05:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 05:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 05:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 05:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 06:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 06:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 06:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 06:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 06:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 06:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 06:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 06:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 06:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 06:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 06:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ857 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ953 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ858 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:29:59 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 1.871533ms >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.89584ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 213.442µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 464.721µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.319778ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 965.288µs >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 950.012µs >[negroni] Started POST /volumes >[heketi] INFO 2018/09/17 07:31:39 Allocating brick set #0 >[negroni] Completed 202 Accepted in 267.217301ms >[asynchttp] INFO 2018/09/17 07:31:39 asynchttp.go:288: Started job 90072a28e339fc3a634b94f97d318275 >[heketi] INFO 2018/09/17 07:31:39 Started async operation: Create Volume >[heketi] INFO 2018/09/17 07:31:39 Creating brick 02e49a43f07335fc6d7261cdbc6d99f3 >[heketi] INFO 2018/09/17 07:31:39 Creating brick a4de903a72ff0eef216a05e723c94bd0 >[heketi] INFO 2018/09/17 07:31:39 Creating brick e044c604b52f6a05c835542b4bf04b01 >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 177.816µs >[kubeexec] DEBUG 2018/09/17 07:31:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir -p /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 >Result: >[kubeexec] DEBUG 2018/09/17 07:31:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir -p /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 >Result: >[kubeexec] DEBUG 2018/09/17 07:31:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir -p /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 195.722µs >[kubeexec] DEBUG 2018/09/17 07:31:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_a4de903a72ff0eef216a05e723c94bd0 --virtualsize 1048576K --name brick_a4de903a72ff0eef216a05e723c94bd0 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_a4de903a72ff0eef216a05e723c94bd0" created. >[kubeexec] DEBUG 2018/09/17 07:31:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b08bde372ba7b620befd7edef290a896/tp_5c3182c3d2766bf38be23b6b9b332f3b --virtualsize 1048576K --name brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_02e49a43f07335fc6d7261cdbc6d99f3" created. >[kubeexec] DEBUG 2018/09/17 07:31:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_e044c604b52f6a05c835542b4bf04b01 --virtualsize 1048576K --name brick_e044c604b52f6a05c835542b4bf04b01 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_e044c604b52f6a05c835542b4bf04b01" created. >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 171.267µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 212.138µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 202.296µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 206.439µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 149.66µs >[kubeexec] DEBUG 2018/09/17 07:31:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_a4de903a72ff0eef216a05e723c94bd0 >Result: meta-data=/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_a4de903a72ff0eef216a05e723c94bd0 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:31:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: awk "BEGIN {print \"/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_a4de903a72ff0eef216a05e723c94bd0 /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:31:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: meta-data=/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_02e49a43f07335fc6d7261cdbc6d99f3 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 124.608µs >[kubeexec] DEBUG 2018/09/17 07:31:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_a4de903a72ff0eef216a05e723c94bd0 /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 >Result: >[kubeexec] DEBUG 2018/09/17 07:31:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: awk "BEGIN {print \"/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_02e49a43f07335fc6d7261cdbc6d99f3 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_e044c604b52f6a05c835542b4bf04b01 >Result: meta-data=/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_e044c604b52f6a05c835542b4bf04b01 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0/brick >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: awk "BEGIN {print \"/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_e044c604b52f6a05c835542b4bf04b01 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_02e49a43f07335fc6d7261cdbc6d99f3 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3/brick >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_e044c604b52f6a05c835542b4bf04b01 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 >Result: >[kubeexec] DEBUG 2018/09/17 07:31:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01/brick >Result: >[cmdexec] INFO 2018/09/17 07:31:47 Creating volume for_testing_of_device_removal replica 3 >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 207.796µs >[kubeexec] DEBUG 2018/09/17 07:31:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume create for_testing_of_device_removal replica 3 10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3/brick 10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0/brick 10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01/brick >Result: volume create: for_testing_of_device_removal: success: please start the volume to access data >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 172.486µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 186.871µs >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 200 OK in 244.406µs >[kubeexec] DEBUG 2018/09/17 07:31:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume start for_testing_of_device_removal >Result: volume start: for_testing_of_device_removal: success >[heketi] INFO 2018/09/17 07:31:51 Create Volume succeeded >[asynchttp] INFO 2018/09/17 07:31:51 asynchttp.go:292: Completed job 90072a28e339fc3a634b94f97d318275 in 11.502571365s >[negroni] Started GET /queue/90072a28e339fc3a634b94f97d318275 >[negroni] Completed 303 See Other in 252.252µs >[negroni] Started GET /volumes/d90d48c096a97ea675023cf1d5f70fe6 >[negroni] Completed 200 OK in 3.89004ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 244.931µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 442.265µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.110236ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 678.535µs >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.173104ms >[heketi] INFO 2018/09/17 07:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130535 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130656 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130525 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:31:59 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 310.515µs >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.116541ms >[negroni] Started GET /volumes/d90d48c096a97ea675023cf1d5f70fe6 >[negroni] Completed 200 OK in 1.023735ms >[negroni] Started DELETE /volumes/d90d48c096a97ea675023cf1d5f70fe6 >[negroni] Completed 202 Accepted in 174.234048ms >[asynchttp] INFO 2018/09/17 07:32:50 asynchttp.go:288: Started job d7c08c7734ff1d02d13828a01ccb329a >[heketi] INFO 2018/09/17 07:32:50 Started async operation: Delete Volume >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 126.036µs >[kubeexec] DEBUG 2018/09/17 07:32:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script snapshot list for_testing_of_device_removal --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> > <snapList> > <count>0</count> > </snapList> ></cliOutput> >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 133.837µs >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 127.31µs >[kubeexec] DEBUG 2018/09/17 07:32:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume stop for_testing_of_device_removal force >Result: volume stop: for_testing_of_device_removal: success >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 136.977µs >[kubeexec] DEBUG 2018/09/17 07:32:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume delete for_testing_of_device_removal >Result: volume delete: for_testing_of_device_removal: success >[heketi] INFO 2018/09/17 07:32:53 Deleting brick 02e49a43f07335fc6d7261cdbc6d99f3 >[heketi] INFO 2018/09/17 07:32:54 Deleting brick a4de903a72ff0eef216a05e723c94bd0 >[heketi] INFO 2018/09/17 07:32:54 Deleting brick e044c604b52f6a05c835542b4bf04b01 >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: umount /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 >Result: >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: umount /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: umount /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 >Result: >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: sed -i.save "/brick_a4de903a72ff0eef216a05e723c94bd0/d" /var/lib/heketi/fstab >Result: >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: sed -i.save "/brick_02e49a43f07335fc6d7261cdbc6d99f3/d" /var/lib/heketi/fstab >Result: >[kubeexec] DEBUG 2018/09/17 07:32:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: sed -i.save "/brick_e044c604b52f6a05c835542b4bf04b01/d" /var/lib/heketi/fstab >Result: >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 227.879µs >[kubeexec] DEBUG 2018/09/17 07:32:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvremove --autobackup=n -f vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 >Result: Logical volume "brick_a4de903a72ff0eef216a05e723c94bd0" successfully removed >[kubeexec] DEBUG 2018/09/17 07:32:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvremove --autobackup=n -f vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: Logical volume "brick_02e49a43f07335fc6d7261cdbc6d99f3" successfully removed >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 458.678µs >[kubeexec] DEBUG 2018/09/17 07:32:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 >Result: Logical volume "brick_e044c604b52f6a05c835542b4bf04b01" successfully removed >[kubeexec] DEBUG 2018/09/17 07:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvs --noheadings --options=thin_count vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_a4de903a72ff0eef216a05e723c94bd0 >Result: 0 >[kubeexec] DEBUG 2018/09/17 07:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvs --noheadings --options=thin_count vg_b08bde372ba7b620befd7edef290a896/tp_5c3182c3d2766bf38be23b6b9b332f3b >Result: 0 >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 113.142µs >[kubeexec] DEBUG 2018/09/17 07:32:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvs --noheadings --options=thin_count vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_e044c604b52f6a05c835542b4bf04b01 >Result: 0 >[kubeexec] DEBUG 2018/09/17 07:32:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvremove --autobackup=n -f vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_a4de903a72ff0eef216a05e723c94bd0 >Result: Logical volume "tp_a4de903a72ff0eef216a05e723c94bd0" successfully removed >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 185.062µs >[kubeexec] DEBUG 2018/09/17 07:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvremove --autobackup=n -f vg_b08bde372ba7b620befd7edef290a896/tp_5c3182c3d2766bf38be23b6b9b332f3b >Result: Logical volume "tp_5c3182c3d2766bf38be23b6b9b332f3b" successfully removed >[kubeexec] DEBUG 2018/09/17 07:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_e044c604b52f6a05c835542b4bf04b01 >Result: Logical volume "tp_e044c604b52f6a05c835542b4bf04b01" successfully removed >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 200 OK in 266.772µs >[kubeexec] DEBUG 2018/09/17 07:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: rmdir /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_a4de903a72ff0eef216a05e723c94bd0 >Result: >[kubeexec] DEBUG 2018/09/17 07:32:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: rmdir /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_02e49a43f07335fc6d7261cdbc6d99f3 >Result: >[kubeexec] DEBUG 2018/09/17 07:32:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: rmdir /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_e044c604b52f6a05c835542b4bf04b01 >Result: >[heketi] INFO 2018/09/17 07:32:59 Delete Volume succeeded >[asynchttp] INFO 2018/09/17 07:32:59 asynchttp.go:292: Completed job d7c08c7734ff1d02d13828a01ccb329a in 8.636619204s >[negroni] Started GET /queue/d7c08c7734ff1d02d13828a01ccb329a >[negroni] Completed 204 No Content in 207.062µs >[negroni] Started POST /volumes >[heketi] INFO 2018/09/17 07:33:04 Allocating brick set #0 >[negroni] Completed 202 Accepted in 154.265724ms >[asynchttp] INFO 2018/09/17 07:33:05 asynchttp.go:288: Started job 9200f0731d1dcdbb21b85138d86e87fd >[heketi] INFO 2018/09/17 07:33:05 Started async operation: Create Volume >[heketi] INFO 2018/09/17 07:33:05 Creating brick c5c810e2ca9fc6a9fa590b20b097a455 >[heketi] INFO 2018/09/17 07:33:05 Creating brick 741a17436fda0156474717169c4a8f7e >[heketi] INFO 2018/09/17 07:33:05 Creating brick a8d68623684b6e915b091d52fcd809e5 >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 174.042µs >[kubeexec] DEBUG 2018/09/17 07:33:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir -p /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e >Result: >[kubeexec] DEBUG 2018/09/17 07:33:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir -p /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir -p /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 130.59µs >[kubeexec] DEBUG 2018/09/17 07:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_741a17436fda0156474717169c4a8f7e --virtualsize 1048576K --name brick_741a17436fda0156474717169c4a8f7e >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_741a17436fda0156474717169c4a8f7e" created. >[kubeexec] DEBUG 2018/09/17 07:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b08bde372ba7b620befd7edef290a896/tp_28d526696ec48c882afcb666194d8df6 --virtualsize 1048576K --name brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_c5c810e2ca9fc6a9fa590b20b097a455" created. >[kubeexec] DEBUG 2018/09/17 07:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_a8d68623684b6e915b091d52fcd809e5 --virtualsize 1048576K --name brick_a8d68623684b6e915b091d52fcd809e5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_a8d68623684b6e915b091d52fcd809e5" created. >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 199.102µs >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 192.779µs >[kubeexec] DEBUG 2018/09/17 07:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_741a17436fda0156474717169c4a8f7e >Result: meta-data=/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_741a17436fda0156474717169c4a8f7e isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: awk "BEGIN {print \"/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_741a17436fda0156474717169c4a8f7e /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: meta-data=/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_c5c810e2ca9fc6a9fa590b20b097a455 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_a8d68623684b6e915b091d52fcd809e5 >Result: meta-data=/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_a8d68623684b6e915b091d52fcd809e5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 123.319µs >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: awk "BEGIN {print \"/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_c5c810e2ca9fc6a9fa590b20b097a455 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: awk "BEGIN {print \"/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_a8d68623684b6e915b091d52fcd809e5 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_741a17436fda0156474717169c4a8f7e /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e/brick >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_a8d68623684b6e915b091d52fcd809e5 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_c5c810e2ca9fc6a9fa590b20b097a455 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5/brick >Result: >[kubeexec] DEBUG 2018/09/17 07:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455/brick >Result: >[cmdexec] INFO 2018/09/17 07:33:09 Creating volume for_testing_of_device_removal replica 3 >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 148.796µs >[kubeexec] DEBUG 2018/09/17 07:33:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume create for_testing_of_device_removal replica 3 10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455/brick 10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e/brick 10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5/brick >Result: volume create: for_testing_of_device_removal: success: please start the volume to access data >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 146.904µs >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 216.425µs >[kubeexec] DEBUG 2018/09/17 07:33:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: gluster --mode=script volume start for_testing_of_device_removal >Result: volume start: for_testing_of_device_removal: success >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 200 OK in 229.025µs >[heketi] INFO 2018/09/17 07:33:13 Create Volume succeeded >[asynchttp] INFO 2018/09/17 07:33:13 asynchttp.go:292: Completed job 9200f0731d1dcdbb21b85138d86e87fd in 8.062307047s >[negroni] Started GET /queue/9200f0731d1dcdbb21b85138d86e87fd >[negroni] Completed 303 See Other in 225.789µs >[negroni] Started GET /volumes/83afa0238a77cf27820f71525e7c66d3 >[negroni] Completed 200 OK in 3.583902ms >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 258.222µs >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.127971ms >[negroni] Started GET /volumes/83afa0238a77cf27820f71525e7c66d3 >[negroni] Completed 200 OK in 1.036044ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 192.625µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 420.01µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.450513ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.085837ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 553.663µs >[negroni] Started POST /volumes >[heketi] INFO 2018/09/17 07:33:48 Allocating brick set #0 >[negroni] Completed 202 Accepted in 207.406697ms >[asynchttp] INFO 2018/09/17 07:33:48 asynchttp.go:288: Started job f392effbb03c48d5cdd962b28530417e >[heketi] INFO 2018/09/17 07:33:48 Started async operation: Create Volume >[heketi] INFO 2018/09/17 07:33:48 Creating brick 83a6d5e8b1072640197ae231f52a48af >[heketi] INFO 2018/09/17 07:33:48 Creating brick 5ba3b90ddb5f4f3923b3f73bf64418d2 >[heketi] INFO 2018/09/17 07:33:48 Creating brick 78021cc4ec9f8219037ecafc49987b69 >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 165.273µs >[kubeexec] DEBUG 2018/09/17 07:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir -p /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir -p /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af >Result: >[kubeexec] DEBUG 2018/09/17 07:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir -p /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 196.003µs >[kubeexec] DEBUG 2018/09/17 07:33:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_b08bde372ba7b620befd7edef290a896/tp_78021cc4ec9f8219037ecafc49987b69 --virtualsize 1048576K --name brick_78021cc4ec9f8219037ecafc49987b69 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_78021cc4ec9f8219037ecafc49987b69" created. >[kubeexec] DEBUG 2018/09/17 07:33:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_ca1fd84ddfbd19783537ea7d61e39f9b/tp_5c46b2de672ea8f555d523b679a4c2d6 --virtualsize 1048576K --name brick_83a6d5e8b1072640197ae231f52a48af >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_83a6d5e8b1072640197ae231f52a48af" created. >[kubeexec] DEBUG 2018/09/17 07:33:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_44ba4eb707b9073f5604f6f9dab91eee/tp_5ba3b90ddb5f4f3923b3f73bf64418d2 --virtualsize 1048576K --name brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_5ba3b90ddb5f4f3923b3f73bf64418d2" created. >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 185.745µs >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 264.812µs >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 221.402µs >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 137.693µs >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 237.196µs >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 208.486µs >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af >Result: meta-data=/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: awk "BEGIN {print \"/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_78021cc4ec9f8219037ecafc49987b69 >Result: meta-data=/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_78021cc4ec9f8219037ecafc49987b69 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_44ba4eb707b9073f5604f6f9dab91eee-brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: meta-data=/dev/mapper/vg_44ba4eb707b9073f5604f6f9dab91eee-brick_5ba3b90ddb5f4f3923b3f73bf64418d2 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: awk "BEGIN {print \"/dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_78021cc4ec9f8219037ecafc49987b69 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: awk "BEGIN {print \"/dev/mapper/vg_44ba4eb707b9073f5604f6f9dab91eee-brick_5ba3b90ddb5f4f3923b3f73bf64418d2 /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af >Result: >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick >Result: >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 183.366µs >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_44ba4eb707b9073f5604f6f9dab91eee-brick_5ba3b90ddb5f4f3923b3f73bf64418d2 /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_b08bde372ba7b620befd7edef290a896-brick_78021cc4ec9f8219037ecafc49987b69 /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69 >Result: >[kubeexec] DEBUG 2018/09/17 07:33:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick >Result: >[kubeexec] DEBUG 2018/09/17 07:33:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: mkdir /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick >Result: >[cmdexec] INFO 2018/09/17 07:33:57 Creating volume for_testing_of_device_removal2 replica 3 >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 185.382µs >[kubeexec] DEBUG 2018/09/17 07:33:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume create for_testing_of_device_removal2 replica 3 10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick 10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick 10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick >Result: volume create: for_testing_of_device_removal2: success: please start the volume to access data >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 204.106µs >[heketi] INFO 2018/09/17 07:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130732 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130927 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 07:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 182.637µs >[kubeexec] DEBUG 2018/09/17 07:34:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume start for_testing_of_device_removal2 >Result: volume start: for_testing_of_device_removal2: success >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 200 OK in 167.587µs >[kubeexec] DEBUG 2018/09/17 07:34:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130877 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:34:01 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:34:01 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:34:01 Create Volume succeeded >[asynchttp] INFO 2018/09/17 07:34:01 asynchttp.go:292: Completed job f392effbb03c48d5cdd962b28530417e in 12.21018209s >[negroni] Started GET /queue/f392effbb03c48d5cdd962b28530417e >[negroni] Completed 303 See Other in 157.436µs >[negroni] Started GET /volumes/ffede5b0a8ce9555bf24fcc762e02b2c >[negroni] Completed 200 OK in 13.565743ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 239.339µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 396.614µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.53317ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.25829ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.201984ms >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 278.789µs >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.101954ms >[negroni] Started GET /volumes/83afa0238a77cf27820f71525e7c66d3 >[negroni] Completed 200 OK in 1.002941ms >[negroni] Started GET /volumes/ffede5b0a8ce9555bf24fcc762e02b2c >[negroni] Completed 200 OK in 583.684µs >[negroni] Started DELETE /volumes/83afa0238a77cf27820f71525e7c66d3 >[negroni] Completed 202 Accepted in 142.67931ms >[asynchttp] INFO 2018/09/17 07:34:30 asynchttp.go:288: Started job 3b2bce5105e27db89519b78d2911d800 >[heketi] INFO 2018/09/17 07:34:30 Started async operation: Delete Volume >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 187.995µs >[kubeexec] DEBUG 2018/09/17 07:34:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script snapshot list for_testing_of_device_removal --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> > <snapList> > <count>0</count> > </snapList> ></cliOutput> >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 202.303µs >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 207.482µs >[kubeexec] DEBUG 2018/09/17 07:34:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume stop for_testing_of_device_removal force >Result: volume stop: for_testing_of_device_removal: success >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 242.636µs >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume delete for_testing_of_device_removal >Result: volume delete: for_testing_of_device_removal: success >[heketi] INFO 2018/09/17 07:34:34 Deleting brick 741a17436fda0156474717169c4a8f7e >[heketi] INFO 2018/09/17 07:34:34 Deleting brick a8d68623684b6e915b091d52fcd809e5 >[heketi] INFO 2018/09/17 07:34:34 Deleting brick c5c810e2ca9fc6a9fa590b20b097a455 >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: umount /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: umount /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e >Result: >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: umount /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 >Result: >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: sed -i.save "/brick_c5c810e2ca9fc6a9fa590b20b097a455/d" /var/lib/heketi/fstab >Result: >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: sed -i.save "/brick_741a17436fda0156474717169c4a8f7e/d" /var/lib/heketi/fstab >Result: >[kubeexec] DEBUG 2018/09/17 07:34:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: sed -i.save "/brick_a8d68623684b6e915b091d52fcd809e5/d" /var/lib/heketi/fstab >Result: >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 231.423µs >[kubeexec] DEBUG 2018/09/17 07:34:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvremove --autobackup=n -f vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: Logical volume "brick_c5c810e2ca9fc6a9fa590b20b097a455" successfully removed >[kubeexec] DEBUG 2018/09/17 07:34:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvremove --autobackup=n -f vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e >Result: Logical volume "brick_741a17436fda0156474717169c4a8f7e" successfully removed >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 178.826µs >[kubeexec] DEBUG 2018/09/17 07:34:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 >Result: Logical volume "brick_a8d68623684b6e915b091d52fcd809e5" successfully removed >[kubeexec] DEBUG 2018/09/17 07:34:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvs --noheadings --options=thin_count vg_b08bde372ba7b620befd7edef290a896/tp_28d526696ec48c882afcb666194d8df6 >Result: 0 >[kubeexec] DEBUG 2018/09/17 07:34:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvs --noheadings --options=thin_count vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_741a17436fda0156474717169c4a8f7e >Result: 0 >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 221.545µs >[kubeexec] DEBUG 2018/09/17 07:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvs --noheadings --options=thin_count vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_a8d68623684b6e915b091d52fcd809e5 >Result: 0 >[kubeexec] DEBUG 2018/09/17 07:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: lvremove --autobackup=n -f vg_b08bde372ba7b620befd7edef290a896/tp_28d526696ec48c882afcb666194d8df6 >Result: Logical volume "tp_28d526696ec48c882afcb666194d8df6" successfully removed >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 188.879µs >[kubeexec] DEBUG 2018/09/17 07:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvremove --autobackup=n -f vg_68c9ef3f5a2e31d2976565f9f187a6cf/tp_741a17436fda0156474717169c4a8f7e >Result: Logical volume "tp_741a17436fda0156474717169c4a8f7e" successfully removed >[kubeexec] DEBUG 2018/09/17 07:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_a8d68623684b6e915b091d52fcd809e5 >Result: Logical volume "tp_a8d68623684b6e915b091d52fcd809e5" successfully removed >[kubeexec] DEBUG 2018/09/17 07:34:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: rmdir /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_c5c810e2ca9fc6a9fa590b20b097a455 >Result: >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 200 OK in 172.066µs >[kubeexec] DEBUG 2018/09/17 07:34:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: rmdir /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_741a17436fda0156474717169c4a8f7e >Result: >[kubeexec] DEBUG 2018/09/17 07:34:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: rmdir /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_a8d68623684b6e915b091d52fcd809e5 >Result: >[heketi] INFO 2018/09/17 07:34:39 Delete Volume succeeded >[asynchttp] INFO 2018/09/17 07:34:39 asynchttp.go:292: Completed job 3b2bce5105e27db89519b78d2911d800 in 8.60740102s >[negroni] Started GET /queue/3b2bce5105e27db89519b78d2911d800 >[negroni] Completed 204 No Content in 232.059µs >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 2.385253ms >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.215458ms >[negroni] Started GET /volumes/ffede5b0a8ce9555bf24fcc762e02b2c >[negroni] Completed 200 OK in 610.969µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 223.266µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 441.465µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.669208ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.125681ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.160064ms >[heketi] INFO 2018/09/17 07:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 07:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:47:59 Cleaned 0 nodes from health cache >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 305.355µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 458.314µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 792.485µs >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.069496ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 682.701µs >[negroni] Started POST /devices/44ba4eb707b9073f5604f6f9dab91eee/state >[negroni] Completed 202 Accepted in 660.777µs >[asynchttp] INFO 2018/09/17 07:49:27 asynchttp.go:288: Started job 97e78e7cfd58d7adfb993732dd1373c6 >[negroni] Started GET /queue/97e78e7cfd58d7adfb993732dd1373c6 >[negroni] Completed 200 OK in 157.516µs >[asynchttp] INFO 2018/09/17 07:49:27 asynchttp.go:292: Completed job 97e78e7cfd58d7adfb993732dd1373c6 in 174.810781ms >[negroni] Started GET /queue/97e78e7cfd58d7adfb993732dd1373c6 >[negroni] Completed 204 No Content in 204.309µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 2.815425ms >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 423.934µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.480623ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 784.626µs >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 768.85µs >[heketi] INFO 2018/09/17 07:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ328 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130955 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:49:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices/44ba4eb707b9073f5604f6f9dab91eee/state >[negroni] Completed 202 Accepted in 358.662µs >[asynchttp] INFO 2018/09/17 07:50:05 asynchttp.go:288: Started job 463e5c26f28ec95d702db3c044b437ef >[heketi] INFO 2018/09/17 07:50:05 Running Remove Device >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 83.378µs >[heketi] INFO 2018/09/17 07:50:05 Replacing brick 5ba3b90ddb5f4f3923b3f73bf64418d2 on device 44ba4eb707b9073f5604f6f9dab91eee on node 216efa66fc7fb3f0516e3a8d0b5aed20 >[cmdexec] INFO 2018/09/17 07:50:05 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ130910 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[kubeexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: gluster --mode=script volume info for_testing_of_device_removal2 --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> > <volInfo> > <volumes> > <volume> > <name>for_testing_of_device_removal2</name> > <id>28cea2b5-6f5b-4341-ab67-45c416f93a8c</id> > <status>1</status> > <statusStr>Started</statusStr> > <snapshotCount>0</snapshotCount> > <brickCount>3</brickCount> > <distCount>3</distCount> > <stripeCount>1</stripeCount> > <replicaCount>3</replicaCount> > <arbiterCount>0</arbiterCount> > <disperseCount>0</disperseCount> > <redundancyCount>0</redundancyCount> > <type>2</type> > <typeStr>Replicate</typeStr> > <transport>0</transport> > <xlators/> > <bricks> > <brick uuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c">10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick<name>10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick</name><hostUuid>d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281">10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick<name>10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick</name><hostUuid>15f88a3a-4bd3-4343-a3da-4a4b88efd281</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="fc918f48-75c7-435c-9ed2-4494074a7b92">10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick<name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick</name><hostUuid>fc918f48-75c7-435c-9ed2-4494074a7b92</hostUuid><isArbiter>0</isArbiter></brick> > </bricks> > <optCount>4</optCount> > <options> > <option> > <name>transport.address-family</name> > <value>inet</value> > </option> > <option> > <name>nfs.disable</name> > <value>on</value> > </option> > <option> > <name>performance.client-io-threads</name> > <value>off</value> > </option> > <option> > <name>cluster.brick-multiplex</name> > <value>on</value> > </option> > </options> > </volume> > <count>1</count> > </volumes> > </volInfo> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:279: {OpRet:0 OpErrno:0 OpErrStr: VolInfo:{XMLName:{Space: Local:volInfo} Volumes:{XMLName:{Space: Local:volumes} Count:1 VolumeList:[{XMLName:{Space: Local:volume} VolumeName:for_testing_of_device_removal2 ID:28cea2b5-6f5b-4341-ab67-45c416f93a8c Status:1 StatusStr:Started BrickCount:3 DistCount:3 StripeCount:1 ReplicaCount:3 ArbiterCount:0 DisperseCount:0 RedundancyCount:0 Type:2 TypeStr:Replicate Transport:0 Bricks:{XMLName:{Space: Local:bricks} BrickList:[{UUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c IsArbiter:0} {UUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 IsArbiter:0} {UUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 IsArbiter:0}]} OptCount:4 Options:{XMLName:{Space: Local:options} OptionList:[{Name:transport.address-family Value:inet} {Name:nfs.disable Value:on} {Name:performance.client-io-threads Value:off} {Name:cluster.brick-multiplex Value:on}]}}]}}} >[kubeexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: gluster --mode=script volume heal for_testing_of_device_removal2 info --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <healInfo> > <bricks> > <brick hostUuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c"> > <name>10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281"> > <name>10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="fc918f48-75c7-435c-9ed2-4494074a7b92"> > <name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > </bricks> > </healInfo> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:395: {OpRet:0 OpErrno:0 OpErrStr: HealInfo:{XMLName:{Space: Local:healInfo} Bricks:{BrickList:[{HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af/brick Status:Connected NumberOfEntries:0} {HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick Status:Connected NumberOfEntries:0} {HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_78021cc4ec9f8219037ecafc49987b69/brick Status:Connected NumberOfEntries:0}]}}} >[heketi] INFO 2018/09/17 07:50:06 Replace brick in brick set BrickSet(3)[83a6d5e8b1072640197ae231f52a48af 5ba3b90ddb5f4f3923b3f73bf64418d2 78021cc4ec9f8219037ecafc49987b69] with index 1 >[heketi] INFO 2018/09/17 07:50:06 Creating brick 23df4c6cf281e27df6058007e2c4a7b5 >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 121.492µs >[kubeexec] DEBUG 2018/09/17 07:50:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir -p /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5 >Result: >[kubeexec] DEBUG 2018/09/17 07:50:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_e9fa7f608e4dacb7d8477ced702b22dd/tp_5f8890165aa7200149d192e712e5ca00 --virtualsize 1048576K --name brick_23df4c6cf281e27df6058007e2c4a7b5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_23df4c6cf281e27df6058007e2c4a7b5" created. >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 187.87µs >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 214.549µs >[kubeexec] DEBUG 2018/09/17 07:50:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_23df4c6cf281e27df6058007e2c4a7b5 >Result: meta-data=/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_23df4c6cf281e27df6058007e2c4a7b5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 119.477µs >[kubeexec] DEBUG 2018/09/17 07:50:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: awk "BEGIN {print \"/dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_23df4c6cf281e27df6058007e2c4a7b5 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:50:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_23df4c6cf281e27df6058007e2c4a7b5 /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5 >Result: >[kubeexec] DEBUG 2018/09/17 07:50:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: mkdir /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5/brick >Result: >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 193.142µs >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 203.286µs >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 187.682µs >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 229.189µs >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 132.544µs >[kubeexec] DEBUG 2018/09/17 07:50:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: gluster --mode=script volume replace-brick for_testing_of_device_removal2 10.70.47.41:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick 10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5/brick commit force >Result: volume replace-brick: success: replace-brick commit force operation successful >[heketi] INFO 2018/09/17 07:50:15 Deleting brick 5ba3b90ddb5f4f3923b3f73bf64418d2 >[kubeexec] DEBUG 2018/09/17 07:50:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: umount /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: >[kubeexec] DEBUG 2018/09/17 07:50:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: sed -i.save "/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/d" /var/lib/heketi/fstab >Result: >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 200 OK in 185.4µs >[kubeexec] DEBUG 2018/09/17 07:50:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: Logical volume "brick_5ba3b90ddb5f4f3923b3f73bf64418d2" successfully removed >[kubeexec] DEBUG 2018/09/17 07:50:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvs --noheadings --options=thin_count vg_44ba4eb707b9073f5604f6f9dab91eee/tp_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: 0 >[kubeexec] DEBUG 2018/09/17 07:50:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: lvremove --autobackup=n -f vg_44ba4eb707b9073f5604f6f9dab91eee/tp_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: Logical volume "tp_5ba3b90ddb5f4f3923b3f73bf64418d2" successfully removed >[kubeexec] DEBUG 2018/09/17 07:50:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: rmdir /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2 >Result: >[heketi] INFO 2018/09/17 07:50:16 replaced brick:5ba3b90ddb5f4f3923b3f73bf64418d2 on node:216efa66fc7fb3f0516e3a8d0b5aed20 at path:/var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee/brick_5ba3b90ddb5f4f3923b3f73bf64418d2/brick with brick:23df4c6cf281e27df6058007e2c4a7b5 on node:216efa66fc7fb3f0516e3a8d0b5aed20 at path:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_23df4c6cf281e27df6058007e2c4a7b5/brick >[asynchttp] INFO 2018/09/17 07:50:16 asynchttp.go:292: Completed job 463e5c26f28ec95d702db3c044b437ef in 10.799234971s >[negroni] Started GET /queue/463e5c26f28ec95d702db3c044b437ef >[negroni] Completed 204 No Content in 121.385µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 221.938µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 462.027µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.856879ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.140184ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.811167ms >[negroni] Started POST /devices/44ba4eb707b9073f5604f6f9dab91eee44ba4eb707b9073f5604f6f9dab91eee/state >[negroni] Completed 404 Not Found in 290.418µs >[negroni] Started POST /devices/44ba4eb707b9073f5604f6f9dab91eee/state >[negroni] Completed 202 Accepted in 434.693µs >[asynchttp] INFO 2018/09/17 07:51:29 asynchttp.go:288: Started job 3c83b5e9105cf04bdbfc3d3409d284d2 >[asynchttp] INFO 2018/09/17 07:51:29 asynchttp.go:292: Completed job 3c83b5e9105cf04bdbfc3d3409d284d2 in 1.99µs >[negroni] Started GET /queue/3c83b5e9105cf04bdbfc3d3409d284d2 >[negroni] Completed 500 Internal Server Error in 89.495µs >[heketi] INFO 2018/09/17 07:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ628 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ701 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ566 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:51:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices/44ba4eb707b9073f5604f6f9dab91eee/state >[negroni] Completed 202 Accepted in 601.794µs >[asynchttp] INFO 2018/09/17 07:53:13 asynchttp.go:288: Started job 2e9e0553b0cc94e6b3458391d978def0 >[asynchttp] INFO 2018/09/17 07:53:13 asynchttp.go:292: Completed job 2e9e0553b0cc94e6b3458391d978def0 in 2.76µs >[negroni] Started GET /queue/2e9e0553b0cc94e6b3458391d978def0 >[negroni] Completed 500 Internal Server Error in 153.233µs >[negroni] Started DELETE /devices/44ba4eb707b9073f5604f6f9dab91eee >[heketi] INFO 2018/09/17 07:53:45 Deleting device 44ba4eb707b9073f5604f6f9dab91eee on node 216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 202 Accepted in 519.913µs >[asynchttp] INFO 2018/09/17 07:53:45 asynchttp.go:288: Started job 8999db08cd1a8ad31d92b56e67134cab >[negroni] Started GET /queue/8999db08cd1a8ad31d92b56e67134cab >[negroni] Completed 200 OK in 81.842µs >[kubeexec] DEBUG 2018/09/17 07:53:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: vgremove -qq vg_44ba4eb707b9073f5604f6f9dab91eee >Result: >[kubeexec] DEBUG 2018/09/17 07:53:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: pvremove -qq '/dev/sde' >Result: >[kubeexec] DEBUG 2018/09/17 07:53:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: ls /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee >Result: >[kubeexec] DEBUG 2018/09/17 07:53:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: rmdir /var/lib/heketi/mounts/vg_44ba4eb707b9073f5604f6f9dab91eee >Result: >[negroni] Started GET /queue/8999db08cd1a8ad31d92b56e67134cab >[negroni] Completed 200 OK in 136.592µs >[heketi] INFO 2018/09/17 07:53:46 Deleted node [44ba4eb707b9073f5604f6f9dab91eee] >[asynchttp] INFO 2018/09/17 07:53:46 asynchttp.go:292: Completed job 8999db08cd1a8ad31d92b56e67134cab in 1.040003421s >[negroni] Started GET /queue/8999db08cd1a8ad31d92b56e67134cab >[negroni] Completed 204 No Content in 201.475µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 3.076467ms >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 488.843µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.715337ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.076724ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.110087ms >[heketi] INFO 2018/09/17 07:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ628 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ701 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ566 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:53:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 07:54:20 Adding device /dev/sde to node 216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 202 Accepted in 135.571958ms >[asynchttp] INFO 2018/09/17 07:54:20 asynchttp.go:288: Started job 5f30f6ebd5a03b440118d78c2aa4d694 >[negroni] Started GET /queue/5f30f6ebd5a03b440118d78c2aa4d694 >[negroni] Completed 200 OK in 296.304µs >[kubeexec] DEBUG 2018/09/17 07:54:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sde' >Result: Physical volume "/dev/sde" successfully created. >[kubeexec] DEBUG 2018/09/17 07:54:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: vgcreate --autobackup=n vg_204812c37ff8293335dd98f2390a1408 /dev/sde >Result: Volume group "vg_204812c37ff8293335dd98f2390a1408" successfully created >[kubeexec] DEBUG 2018/09/17 07:54:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: vgdisplay -c vg_204812c37ff8293335dd98f2390a1408 >Result: vg_204812c37ff8293335dd98f2390a1408:r/w:772:-1:0:0:0:-1:0:1:1:209580032:4096:51167:0:51167:AXJXDL-54c8-emTs-vadm-UkXT-RwSa-yRrLW4 >[cmdexec] DEBUG 2018/09/17 07:54:21 /src/github.com/heketi/heketi/executors/cmdexec/device.go:161: /dev/sde in vp-ansible-v310-ga-app-cns-0 has TotalSize:209580032, FreeSize:209580032, UsedSize:0 >[heketi] INFO 2018/09/17 07:54:21 Added device /dev/sde >[asynchttp] INFO 2018/09/17 07:54:21 asynchttp.go:292: Completed job 5f30f6ebd5a03b440118d78c2aa4d694 in 827.369347ms >[negroni] Started GET /queue/5f30f6ebd5a03b440118d78c2aa4d694 >[negroni] Completed 204 No Content in 149.801µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 2.777665ms >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 460.262µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.768825ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.185431ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 897.195µs >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 07:54:59 Adding device /dev/sdf to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 150.334981ms >[asynchttp] INFO 2018/09/17 07:54:59 asynchttp.go:288: Started job 9d2106c18db16e8005d8393596ae1d86 >[negroni] Started GET /queue/9d2106c18db16e8005d8393596ae1d86 >[negroni] Completed 200 OK in 171.459µs >[negroni] Started GET /queue/9d2106c18db16e8005d8393596ae1d86 >[negroni] Completed 200 OK in 222.292µs >[kubeexec] DEBUG 2018/09/17 07:55:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf' >Result: Physical volume "/dev/sdf" successfully created. >[kubeexec] DEBUG 2018/09/17 07:55:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: vgcreate --autobackup=n vg_1517317e9a1fc96c830c9493c49833f9 /dev/sdf >Result: Volume group "vg_1517317e9a1fc96c830c9493c49833f9" successfully created >[kubeexec] DEBUG 2018/09/17 07:55:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: vgdisplay -c vg_1517317e9a1fc96c830c9493c49833f9 >Result: vg_1517317e9a1fc96c830c9493c49833f9:r/w:772:-1:0:0:0:-1:0:1:1:104722432:4096:25567:0:25567:bCRJFU-J55C-wZsF-WAoj-CcPr-P2sc-h8vI3U >[cmdexec] DEBUG 2018/09/17 07:55:00 /src/github.com/heketi/heketi/executors/cmdexec/device.go:161: /dev/sdf in vp-ansible-v310-ga-app-cns-1 has TotalSize:104722432, FreeSize:104722432, UsedSize:0 >[heketi] INFO 2018/09/17 07:55:00 Added device /dev/sdf >[asynchttp] INFO 2018/09/17 07:55:00 asynchttp.go:292: Completed job 9d2106c18db16e8005d8393596ae1d86 in 1.541842374s >[negroni] Started GET /queue/9d2106c18db16e8005d8393596ae1d86 >[negroni] Completed 204 No Content in 208.772µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 1.838871ms >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 269.437µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.72608ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.159924ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.2192ms >[negroni] Started POST /devices/68c9ef3f5a2e31d2976565f9f187a6cf/state >[negroni] Completed 202 Accepted in 685.51µs >[asynchttp] INFO 2018/09/17 07:55:42 asynchttp.go:288: Started job 83b9c311c7e1f069c65f8fc3a8d3d2a6 >[negroni] Started GET /queue/83b9c311c7e1f069c65f8fc3a8d3d2a6 >[negroni] Completed 200 OK in 194.632µs >[asynchttp] INFO 2018/09/17 07:55:42 asynchttp.go:292: Completed job 83b9c311c7e1f069c65f8fc3a8d3d2a6 in 161.082575ms >[negroni] Started GET /queue/83b9c311c7e1f069c65f8fc3a8d3d2a6 >[negroni] Completed 204 No Content in 267.975µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 2.806658ms >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 730.999µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 2.855908ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.184043ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.279325ms >[heketi] INFO 2018/09/17 07:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ628 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ701 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ566 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:55:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices/68c9ef3f5a2e31d2976565f9f187a6cf/state >[negroni] Completed 202 Accepted in 575.123µs >[asynchttp] INFO 2018/09/17 07:56:34 asynchttp.go:288: Started job 5861f3b65499df8a1d7241598e771f31 >[heketi] INFO 2018/09/17 07:56:34 Running Remove Device >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 159.463µs >[heketi] INFO 2018/09/17 07:56:34 Replacing brick e04cc4729e00e4ec21aa1b028651d599 on device 68c9ef3f5a2e31d2976565f9f187a6cf on node 8966bb38131b92389d55e09862ca9cce >[cmdexec] INFO 2018/09/17 07:56:34 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:56:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ566 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[kubeexec] DEBUG 2018/09/17 07:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume info heketidbstorage --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> > <volInfo> > <volumes> > <volume> > <name>heketidbstorage</name> > <id>05199f2c-a3ab-4a0b-9086-699682f1fdaf</id> > <status>1</status> > <statusStr>Started</statusStr> > <snapshotCount>0</snapshotCount> > <brickCount>3</brickCount> > <distCount>3</distCount> > <stripeCount>1</stripeCount> > <replicaCount>3</replicaCount> > <arbiterCount>0</arbiterCount> > <disperseCount>0</disperseCount> > <redundancyCount>0</redundancyCount> > <type>2</type> > <typeStr>Replicate</typeStr> > <transport>0</transport> > <xlators/> > <bricks> > <brick uuid="fc918f48-75c7-435c-9ed2-4494074a7b92">10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick<name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick</name><hostUuid>fc918f48-75c7-435c-9ed2-4494074a7b92</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c">10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick<name>10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick</name><hostUuid>d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281">10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick<name>10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick</name><hostUuid>15f88a3a-4bd3-4343-a3da-4a4b88efd281</hostUuid><isArbiter>0</isArbiter></brick> > </bricks> > <optCount>4</optCount> > <options> > <option> > <name>transport.address-family</name> > <value>inet</value> > </option> > <option> > <name>nfs.disable</name> > <value>on</value> > </option> > <option> > <name>performance.client-io-threads</name> > <value>off</value> > </option> > <option> > <name>cluster.brick-multiplex</name> > <value>on</value> > </option> > </options> > </volume> > <count>1</count> > </volumes> > </volInfo> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 07:56:35 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:279: {OpRet:0 OpErrno:0 OpErrStr: VolInfo:{XMLName:{Space: Local:volInfo} Volumes:{XMLName:{Space: Local:volumes} Count:1 VolumeList:[{XMLName:{Space: Local:volume} VolumeName:heketidbstorage ID:05199f2c-a3ab-4a0b-9086-699682f1fdaf Status:1 StatusStr:Started BrickCount:3 DistCount:3 StripeCount:1 ReplicaCount:3 ArbiterCount:0 DisperseCount:0 RedundancyCount:0 Type:2 TypeStr:Replicate Transport:0 Bricks:{XMLName:{Space: Local:bricks} BrickList:[{UUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 IsArbiter:0} {UUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c IsArbiter:0} {UUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 IsArbiter:0}]} OptCount:4 Options:{XMLName:{Space: Local:options} OptionList:[{Name:transport.address-family Value:inet} {Name:nfs.disable Value:on} {Name:performance.client-io-threads Value:off} {Name:cluster.brick-multiplex Value:on}]}}]}}} >[kubeexec] DEBUG 2018/09/17 07:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume heal heketidbstorage info --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <healInfo> > <bricks> > <brick hostUuid="fc918f48-75c7-435c-9ed2-4494074a7b92"> > <name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c"> > <name>10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281"> > <name>10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > </bricks> > </healInfo> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 07:56:35 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:395: {OpRet:0 OpErrno:0 OpErrStr: HealInfo:{XMLName:{Space: Local:healInfo} Bricks:{BrickList:[{HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick Status:Connected NumberOfEntries:0} {HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick Status:Connected NumberOfEntries:0} {HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick Status:Connected NumberOfEntries:0}]}}} >[heketi] INFO 2018/09/17 07:56:35 Replace brick in brick set BrickSet(3)[41234f2611768363d4dad45fc66c340f e04cc4729e00e4ec21aa1b028651d599 862dfad9e7e06fde3aa0972bf89b9bdf] with index 1 >[heketi] INFO 2018/09/17 07:56:35 Creating brick 651d564bd98fb500018dadbe35ee60b0 >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 225.559µs >[kubeexec] DEBUG 2018/09/17 07:56:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir -p /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0 >Result: >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 149.841µs >[kubeexec] DEBUG 2018/09/17 07:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_1517317e9a1fc96c830c9493c49833f9/tp_ff12ee2104892099503b087db5b8aefd --virtualsize 2097152K --name brick_651d564bd98fb500018dadbe35ee60b0 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_651d564bd98fb500018dadbe35ee60b0" created. >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 213.253µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 147.985µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 240.804µs >[kubeexec] DEBUG 2018/09/17 07:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 >Result: meta-data=/dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 isize=512 agcount=8, agsize=65536 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=524288, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 07:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: awk "BEGIN {print \"/dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 07:56:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0 >Result: >[kubeexec] DEBUG 2018/09/17 07:56:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick >Result: >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 137.121µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 313.305µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 296.202µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 260.235µs >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 205.079µs >[kubeexec] DEBUG 2018/09/17 07:56:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume replace-brick heketidbstorage 10.70.46.45:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick 10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick commit force >Result: volume replace-brick: success: replace-brick commit force operation successful >[heketi] INFO 2018/09/17 07:56:45 Deleting brick e04cc4729e00e4ec21aa1b028651d599 >[kubeexec] ERROR 2018/09/17 07:56:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [umount /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599] on glusterfs-cns-6hrtj: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >] >[cmdexec] ERROR 2018/09/17 07:56:45 /src/github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount: /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >[kubeexec] DEBUG 2018/09/17 07:56:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount >Result: overlay on / type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/46SFU5GNIJOUOLZ3ZI5SZZZMDI:/var/lib/docker/overlay2/l/YIPVLRVJ2UXSMVNWGSJ74F74GC:/var/lib/docker/overlay2/l/CB4CLASTA4L5NTPE5OL6LTFR57:/var/lib/docker/overlay2/l/Y7WYUT7DBAZQ5I5ZYOOJXBBBWK,upperdir=/var/lib/docker/overlay2/17da7225394ecce852a7f44df876ef6e7416a1a96fd393c602a2599808cc4bb8/diff,workdir=/var/lib/docker/overlay2/17da7225394ecce852a7f44df876ef6e7416a1a96fd393c602a2599808cc4bb8/work) >proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) >sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) >/dev/sdc on /run type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=16369948k,nr_inodes=4092487,mode=755) >tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) >devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) >mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel) >hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) >/dev/mapper/docker--vol-dockerlv on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/target type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/sdc on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /run/lvm type tmpfs (rw,nosuid,nodev,seclabel,mode=755) >shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c103,c148",size=65536k) >/dev/sdc on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/ssl type xfs (ro,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/docker--vol-dockerlv on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/docker--vol-dockerlv on /run/secrets type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/rhel_dhcp46--210-root on /usr/lib/modules type xfs (ro,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/glusterd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/heketi type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755) >cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) >cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory) >cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu) >cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio) >cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices) >cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids) >cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset) >cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls) >cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb) >cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer) >cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event) >/dev/mapper/rhel_dhcp46--210-root on /var/log/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/misc/glusterfsd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel) >systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=91418) >debugfs on /sys/kernel/debug type debugfs (rw,relatime) >configfs on /sys/kernel/config type configfs (rw,relatime) >sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) >/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599 on /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af on /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >/dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 on /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >[heketi] ERROR 2018/09/17 07:56:45 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:387: Error destroying old brick: umount: /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >[kubeexec] DEBUG 2018/09/17 07:56:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: sed -i.save "/brick_e04cc4729e00e4ec21aa1b028651d599/d" /var/lib/heketi/fstab >Result: >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 200 OK in 137.882µs >[heketi] INFO 2018/09/17 07:56:45 replaced brick:e04cc4729e00e4ec21aa1b028651d599 on node:8966bb38131b92389d55e09862ca9cce at path:/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick with brick:651d564bd98fb500018dadbe35ee60b0 on node:8966bb38131b92389d55e09862ca9cce at path:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick >[asynchttp] INFO 2018/09/17 07:56:45 asynchttp.go:292: Completed job 5861f3b65499df8a1d7241598e771f31 in 11.316324717s >[negroni] Started GET /queue/5861f3b65499df8a1d7241598e771f31 >[negroni] Completed 204 No Content in 192.662µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 202.879µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 409.478µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 3.478039ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.22497ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.369606ms >[heketi] INFO 2018/09/17 07:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:57:59 Cleaned 0 nodes from health cache >[negroni] Started DELETE /devices/68c9ef3f5a2e31d2976565f9f187a6cf >[heketi] INFO 2018/09/17 07:58:30 Deleting device 68c9ef3f5a2e31d2976565f9f187a6cf on node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 464.171µs >[asynchttp] INFO 2018/09/17 07:58:30 asynchttp.go:288: Started job b8b0eac32d7e5ad180bf71b004bcf8c2 >[negroni] Started GET /queue/b8b0eac32d7e5ad180bf71b004bcf8c2 >[negroni] Completed 200 OK in 87.285µs >[kubeexec] ERROR 2018/09/17 07:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [vgremove -qq vg_68c9ef3f5a2e31d2976565f9f187a6cf] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [Do you really want to remove volume group "vg_68c9ef3f5a2e31d2976565f9f187a6cf" containing 2 logical volumes? [y/n]: [n] > Volume group "vg_68c9ef3f5a2e31d2976565f9f187a6cf" not removed >] >[cmdexec] ERROR 2018/09/17 07:58:30 /src/github.com/heketi/heketi/executors/cmdexec/device.go:83: Error while deleting device /dev/sdd with id 68c9ef3f5a2e31d2976565f9f187a6cf on host vp-ansible-v310-ga-app-cns-1: Do you really want to remove volume group "vg_68c9ef3f5a2e31d2976565f9f187a6cf" containing 2 logical volumes? [y/n]: [n] > Volume group "vg_68c9ef3f5a2e31d2976565f9f187a6cf" not removed >[kubeexec] DEBUG 2018/09/17 07:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: ls /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf >Result: brick_e04cc4729e00e4ec21aa1b028651d599 >[kubeexec] ERROR 2018/09/17 07:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [rmdir /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf] on glusterfs-cns-6hrtj: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf': Directory not empty >] >[cmdexec] ERROR 2018/09/17 07:58:30 /src/github.com/heketi/heketi/executors/cmdexec/device.go:102: Error while removing the VG directory >[heketi] INFO 2018/09/17 07:58:30 Deleted node [68c9ef3f5a2e31d2976565f9f187a6cf] >[asynchttp] INFO 2018/09/17 07:58:30 asynchttp.go:292: Completed job b8b0eac32d7e5ad180bf71b004bcf8c2 in 526.391902ms >[negroni] Started GET /queue/b8b0eac32d7e5ad180bf71b004bcf8c2 >[negroni] Completed 204 No Content in 202.333µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 236.376µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 451.258µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.125021ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.136868ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.127431ms >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 07:59:14 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 136.847044ms >[asynchttp] INFO 2018/09/17 07:59:14 asynchttp.go:288: Started job 16f71d4bb31bff20a4639631cb9e58a4 >[negroni] Started GET /queue/16f71d4bb31bff20a4639631cb9e58a4 >[negroni] Completed 200 OK in 174.389µs >[kubeexec] ERROR 2018/09/17 07:59:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 07:59:15 asynchttp.go:292: Completed job 16f71d4bb31bff20a4639631cb9e58a4 in 597.697858ms >[negroni] Started GET /queue/16f71d4bb31bff20a4639631cb9e58a4 >[negroni] Completed 500 Internal Server Error in 235.872µs >[heketi] INFO 2018/09/17 07:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 07:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 07:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 07:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 07:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 07:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 07:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 07:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 07:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 07:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 07:59:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:00:06 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 238.369604ms >[asynchttp] INFO 2018/09/17 08:00:06 asynchttp.go:288: Started job 17d6193945dddc7c7039e01ce9b6cdf1 >[negroni] Started GET /queue/17d6193945dddc7c7039e01ce9b6cdf1 >[negroni] Completed 200 OK in 150.188µs >[kubeexec] ERROR 2018/09/17 08:00:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:00:06 asynchttp.go:292: Completed job 17d6193945dddc7c7039e01ce9b6cdf1 in 500.850212ms >[negroni] Started GET /queue/17d6193945dddc7c7039e01ce9b6cdf1 >[negroni] Completed 500 Internal Server Error in 226.845µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 231.066µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 261.716µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.146761ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 637.194µs >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.323156ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 251.593µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 428.511µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.136181ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.135127ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.11835ms >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:01:06 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 194.69726ms >[asynchttp] INFO 2018/09/17 08:01:06 asynchttp.go:288: Started job 9899b6b625a6b2aff8ae6b6d4e7d6f03 >[negroni] Started GET /queue/9899b6b625a6b2aff8ae6b6d4e7d6f03 >[negroni] Completed 200 OK in 201.786µs >[kubeexec] ERROR 2018/09/17 08:01:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:01:07 asynchttp.go:292: Completed job 9899b6b625a6b2aff8ae6b6d4e7d6f03 in 550.13633ms >[negroni] Started GET /queue/9899b6b625a6b2aff8ae6b6d4e7d6f03 >[negroni] Completed 500 Internal Server Error in 229.525µs >[heketi] INFO 2018/09/17 08:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 08:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:01:59 Cleaned 0 nodes from health cache >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.198101ms >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:02:36 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 214.672217ms >[asynchttp] INFO 2018/09/17 08:02:36 asynchttp.go:288: Started job 5afe7096c2e6807d5688064eda052650 >[negroni] Started GET /queue/5afe7096c2e6807d5688064eda052650 >[negroni] Completed 200 OK in 146.038µs >[kubeexec] ERROR 2018/09/17 08:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:02:37 asynchttp.go:292: Completed job 5afe7096c2e6807d5688064eda052650 in 511.886073ms >[negroni] Started GET /queue/5afe7096c2e6807d5688064eda052650 >[negroni] Completed 500 Internal Server Error in 253.505µs >[heketi] INFO 2018/09/17 08:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 08:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:03:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:05:06 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 134.862458ms >[asynchttp] INFO 2018/09/17 08:05:06 asynchttp.go:288: Started job 25672e1b689af25bda406d4bdda6ebe4 >[negroni] Started GET /queue/25672e1b689af25bda406d4bdda6ebe4 >[negroni] Completed 200 OK in 126.484µs >[kubeexec] ERROR 2018/09/17 08:05:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:05:06 asynchttp.go:292: Completed job 25672e1b689af25bda406d4bdda6ebe4 in 543.325195ms >[negroni] Started GET /queue/25672e1b689af25bda406d4bdda6ebe4 >[negroni] Completed 500 Internal Server Error in 154.926µs >[heketi] INFO 2018/09/17 08:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 08:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 08:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:07:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:09:03 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 203.177203ms >[asynchttp] INFO 2018/09/17 08:09:03 asynchttp.go:288: Started job 95c1ef1a8ac278992a526d4b80330af1 >[negroni] Started GET /queue/95c1ef1a8ac278992a526d4b80330af1 >[negroni] Completed 200 OK in 191.389µs >[kubeexec] ERROR 2018/09/17 08:09:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:09:04 asynchttp.go:292: Completed job 95c1ef1a8ac278992a526d4b80330af1 in 484.829774ms >[negroni] Started GET /queue/95c1ef1a8ac278992a526d4b80330af1 >[negroni] Completed 500 Internal Server Error in 212.669µs >[heketi] INFO 2018/09/17 08:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ882 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ833 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 > ââ932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 >[heketi] INFO 2018/09/17 08:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:09:59 Cleaned 0 nodes from health cache >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 258.155µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 476.831µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.17654ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 771.969µs >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.138044ms >[negroni] Started POST /devices/1517317e9a1fc96c830c9493c49833f9/state >[negroni] Completed 202 Accepted in 659.799µs >[asynchttp] INFO 2018/09/17 08:10:23 asynchttp.go:288: Started job 5461beb4ebeb24bb43854bf8bb8b8e73 >[negroni] Started GET /queue/5461beb4ebeb24bb43854bf8bb8b8e73 >[negroni] Completed 200 OK in 184.892µs >[asynchttp] INFO 2018/09/17 08:10:24 asynchttp.go:292: Completed job 5461beb4ebeb24bb43854bf8bb8b8e73 in 364.483213ms >[negroni] Started GET /queue/5461beb4ebeb24bb43854bf8bb8b8e73 >[negroni] Completed 204 No Content in 199.373µs >[negroni] Started POST /devices/1517317e9a1fc96c830c9493c49833f9/state >[negroni] Completed 202 Accepted in 596.797µs >[asynchttp] INFO 2018/09/17 08:10:33 asynchttp.go:288: Started job b40de5a3727be052f1d1acfa9a6bfbaf >[heketi] INFO 2018/09/17 08:10:33 Running Remove Device >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 144.702µs >[heketi] INFO 2018/09/17 08:10:33 Replacing brick 651d564bd98fb500018dadbe35ee60b0 on device 1517317e9a1fc96c830c9493c49833f9 on node 8966bb38131b92389d55e09862ca9cce >[cmdexec] INFO 2018/09/17 08:10:33 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:10:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ881 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[kubeexec] DEBUG 2018/09/17 08:10:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume info heketidbstorage --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> > <volInfo> > <volumes> > <volume> > <name>heketidbstorage</name> > <id>05199f2c-a3ab-4a0b-9086-699682f1fdaf</id> > <status>1</status> > <statusStr>Started</statusStr> > <snapshotCount>0</snapshotCount> > <brickCount>3</brickCount> > <distCount>3</distCount> > <stripeCount>1</stripeCount> > <replicaCount>3</replicaCount> > <arbiterCount>0</arbiterCount> > <disperseCount>0</disperseCount> > <redundancyCount>0</redundancyCount> > <type>2</type> > <typeStr>Replicate</typeStr> > <transport>0</transport> > <xlators/> > <bricks> > <brick uuid="fc918f48-75c7-435c-9ed2-4494074a7b92">10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick<name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick</name><hostUuid>fc918f48-75c7-435c-9ed2-4494074a7b92</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c">10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick<name>10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick</name><hostUuid>d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c</hostUuid><isArbiter>0</isArbiter></brick> > <brick uuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281">10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick<name>10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick</name><hostUuid>15f88a3a-4bd3-4343-a3da-4a4b88efd281</hostUuid><isArbiter>0</isArbiter></brick> > </bricks> > <optCount>4</optCount> > <options> > <option> > <name>transport.address-family</name> > <value>inet</value> > </option> > <option> > <name>nfs.disable</name> > <value>on</value> > </option> > <option> > <name>performance.client-io-threads</name> > <value>off</value> > </option> > <option> > <name>cluster.brick-multiplex</name> > <value>on</value> > </option> > </options> > </volume> > <count>1</count> > </volumes> > </volInfo> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 08:10:33 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:279: {OpRet:0 OpErrno:0 OpErrStr: VolInfo:{XMLName:{Space: Local:volInfo} Volumes:{XMLName:{Space: Local:volumes} Count:1 VolumeList:[{XMLName:{Space: Local:volume} VolumeName:heketidbstorage ID:05199f2c-a3ab-4a0b-9086-699682f1fdaf Status:1 StatusStr:Started BrickCount:3 DistCount:3 StripeCount:1 ReplicaCount:3 ArbiterCount:0 DisperseCount:0 RedundancyCount:0 Type:2 TypeStr:Replicate Transport:0 Bricks:{XMLName:{Space: Local:bricks} BrickList:[{UUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 IsArbiter:0} {UUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c IsArbiter:0} {UUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 IsArbiter:0}]} OptCount:4 Options:{XMLName:{Space: Local:options} OptionList:[{Name:transport.address-family Value:inet} {Name:nfs.disable Value:on} {Name:performance.client-io-threads Value:off} {Name:cluster.brick-multiplex Value:on}]}}]}}} >[kubeexec] DEBUG 2018/09/17 08:10:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume heal heketidbstorage info --xml >Result: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> ><cliOutput> > <healInfo> > <bricks> > <brick hostUuid="fc918f48-75c7-435c-9ed2-4494074a7b92"> > <name>10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c"> > <name>10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > <brick hostUuid="15f88a3a-4bd3-4343-a3da-4a4b88efd281"> > <name>10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick</name> > <status>Connected</status> > <numberOfEntries>0</numberOfEntries> > </brick> > </bricks> > </healInfo> > <opRet>0</opRet> > <opErrno>0</opErrno> > <opErrstr/> ></cliOutput> >[cmdexec] DEBUG 2018/09/17 08:10:33 /src/github.com/heketi/heketi/executors/cmdexec/volume.go:395: {OpRet:0 OpErrno:0 OpErrStr: HealInfo:{XMLName:{Space: Local:healInfo} Bricks:{BrickList:[{HostUUID:fc918f48-75c7-435c-9ed2-4494074a7b92 Name:10.70.46.57:/var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick Status:Connected NumberOfEntries:0} {HostUUID:d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c Name:10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick Status:Connected NumberOfEntries:0} {HostUUID:15f88a3a-4bd3-4343-a3da-4a4b88efd281 Name:10.70.47.41:/var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick Status:Connected NumberOfEntries:0}]}}} >[heketi] INFO 2018/09/17 08:10:33 Replace brick in brick set BrickSet(3)[41234f2611768363d4dad45fc66c340f 651d564bd98fb500018dadbe35ee60b0 862dfad9e7e06fde3aa0972bf89b9bdf] with index 1 >[heketi] INFO 2018/09/17 08:10:34 Creating brick aea6efb6d0301b9b35b66e3b7821606e >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 217.072µs >[kubeexec] DEBUG 2018/09/17 08:10:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir -p /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e >Result: >[kubeexec] DEBUG 2018/09/17 08:10:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: lvcreate --autobackup=n --poolmetadatasize 12288K --chunksize 256K --size 2097152K --thin vg_ca1fd84ddfbd19783537ea7d61e39f9b/tp_534b36a164af13c58dd8d4f14368e7f0 --virtualsize 2097152K --name brick_aea6efb6d0301b9b35b66e3b7821606e >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_aea6efb6d0301b9b35b66e3b7821606e" created. >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 221.966µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 225.385µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 135.167µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 197.759µs >[kubeexec] DEBUG 2018/09/17 08:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_aea6efb6d0301b9b35b66e3b7821606e >Result: meta-data=/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_aea6efb6d0301b9b35b66e3b7821606e isize=512 agcount=8, agsize=65536 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=524288, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/09/17 08:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: awk "BEGIN {print \"/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_aea6efb6d0301b9b35b66e3b7821606e /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/09/17 08:10:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_aea6efb6d0301b9b35b66e3b7821606e /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e >Result: >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 199.009µs >[kubeexec] DEBUG 2018/09/17 08:10:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mkdir /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e/brick >Result: >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 209.766µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 250.498µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 151.798µs >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 629.697µs >[kubeexec] DEBUG 2018/09/17 08:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: gluster --mode=script volume replace-brick heketidbstorage 10.70.46.45:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick 10.70.46.45:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e/brick commit force >Result: volume replace-brick: success: replace-brick commit force operation successful >[heketi] INFO 2018/09/17 08:10:44 Deleting brick 651d564bd98fb500018dadbe35ee60b0 >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 200 OK in 187.206µs >[kubeexec] ERROR 2018/09/17 08:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [umount /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0] on glusterfs-cns-6hrtj: Err[command terminated with exit code 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >] >[cmdexec] ERROR 2018/09/17 08:10:44 /src/github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount: /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >[kubeexec] DEBUG 2018/09/17 08:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: mount >Result: overlay on / type overlay (rw,relatime,seclabel,lowerdir=/var/lib/docker/overlay2/l/46SFU5GNIJOUOLZ3ZI5SZZZMDI:/var/lib/docker/overlay2/l/YIPVLRVJ2UXSMVNWGSJ74F74GC:/var/lib/docker/overlay2/l/CB4CLASTA4L5NTPE5OL6LTFR57:/var/lib/docker/overlay2/l/Y7WYUT7DBAZQ5I5ZYOOJXBBBWK,upperdir=/var/lib/docker/overlay2/17da7225394ecce852a7f44df876ef6e7416a1a96fd393c602a2599808cc4bb8/diff,workdir=/var/lib/docker/overlay2/17da7225394ecce852a7f44df876ef6e7416a1a96fd393c602a2599808cc4bb8/work) >proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) >sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) >/dev/sdc on /run type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=16369948k,nr_inodes=4092487,mode=755) >tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) >devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) >mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel) >hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) >/dev/mapper/docker--vol-dockerlv on /etc/resolv.conf type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/target type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/sdc on /dev/termination-log type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /run/lvm type tmpfs (rw,nosuid,nodev,seclabel,mode=755) >shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,context="system_u:object_r:container_file_t:s0:c103,c148",size=65536k) >/dev/sdc on /etc/hosts type xfs (rw,relatime,seclabel,attr2,inode64,grpquota) >/dev/mapper/rhel_dhcp46--210-root on /etc/ssl type xfs (ro,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/docker--vol-dockerlv on /etc/hostname type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/docker--vol-dockerlv on /run/secrets type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) >/dev/mapper/rhel_dhcp46--210-root on /usr/lib/modules type xfs (ro,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/glusterd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/heketi type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755) >cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) >cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory) >cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu) >cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio) >cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices) >cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids) >cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset) >cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls) >cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb) >cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer) >cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event) >/dev/mapper/rhel_dhcp46--210-root on /var/log/glusterfs type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >/dev/mapper/rhel_dhcp46--210-root on /var/lib/misc/glusterfsd type xfs (rw,relatime,seclabel,attr2,inode64,noquota) >tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime,seclabel) >systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=23,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=91418) >debugfs on /sys/kernel/debug type debugfs (rw,relatime) >configfs on /sys/kernel/config type configfs (rw,relatime) >sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) >/dev/mapper/vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599 on /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_83a6d5e8b1072640197ae231f52a48af on /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_83a6d5e8b1072640197ae231f52a48af type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >/dev/mapper/vg_1517317e9a1fc96c830c9493c49833f9-brick_651d564bd98fb500018dadbe35ee60b0 on /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0 type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >/dev/mapper/vg_ca1fd84ddfbd19783537ea7d61e39f9b-brick_aea6efb6d0301b9b35b66e3b7821606e on /var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e type xfs (rw,noatime,seclabel,nouuid,attr2,inode64,logbsize=256k,sunit=512,swidth=512,noquota) >[heketi] ERROR 2018/09/17 08:10:44 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:387: Error destroying old brick: umount: /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0: target is busy. > (In some cases useful info about processes that use > the device is found by lsof(8) or fuser(1)) >[kubeexec] DEBUG 2018/09/17 08:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: sed -i.save "/brick_651d564bd98fb500018dadbe35ee60b0/d" /var/lib/heketi/fstab >Result: >[heketi] INFO 2018/09/17 08:10:44 replaced brick:651d564bd98fb500018dadbe35ee60b0 on node:8966bb38131b92389d55e09862ca9cce at path:/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9/brick_651d564bd98fb500018dadbe35ee60b0/brick with brick:aea6efb6d0301b9b35b66e3b7821606e on node:8966bb38131b92389d55e09862ca9cce at path:/var/lib/heketi/mounts/vg_ca1fd84ddfbd19783537ea7d61e39f9b/brick_aea6efb6d0301b9b35b66e3b7821606e/brick >[asynchttp] INFO 2018/09/17 08:10:44 asynchttp.go:292: Completed job b40de5a3727be052f1d1acfa9a6bfbaf in 11.807412873s >[negroni] Started GET /queue/b40de5a3727be052f1d1acfa9a6bfbaf >[negroni] Completed 204 No Content in 207.076µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 232.479µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 422.081µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.156041ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.093761ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.289026ms >[negroni] Started DELETE /devices/1517317e9a1fc96c830c9493c49833f9 >[heketi] INFO 2018/09/17 08:11:04 Deleting device 1517317e9a1fc96c830c9493c49833f9 on node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 482.711µs >[asynchttp] INFO 2018/09/17 08:11:04 asynchttp.go:288: Started job 0984613fd7fc475ff5b48e5eddf9cd67 >[negroni] Started GET /queue/0984613fd7fc475ff5b48e5eddf9cd67 >[negroni] Completed 200 OK in 151.445µs >[kubeexec] ERROR 2018/09/17 08:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [vgremove -qq vg_1517317e9a1fc96c830c9493c49833f9] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [Do you really want to remove volume group "vg_1517317e9a1fc96c830c9493c49833f9" containing 2 logical volumes? [y/n]: [n] > Volume group "vg_1517317e9a1fc96c830c9493c49833f9" not removed >] >[cmdexec] ERROR 2018/09/17 08:11:04 /src/github.com/heketi/heketi/executors/cmdexec/device.go:83: Error while deleting device /dev/sdf with id 1517317e9a1fc96c830c9493c49833f9 on host vp-ansible-v310-ga-app-cns-1: Do you really want to remove volume group "vg_1517317e9a1fc96c830c9493c49833f9" containing 2 logical volumes? [y/n]: [n] > Volume group "vg_1517317e9a1fc96c830c9493c49833f9" not removed >[kubeexec] DEBUG 2018/09/17 08:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: ls /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9 >Result: brick_651d564bd98fb500018dadbe35ee60b0 >[kubeexec] ERROR 2018/09/17 08:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [rmdir /var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9] on glusterfs-cns-6hrtj: Err[command terminated with exit code 1]: Stdout []: Stderr [rmdir: failed to remove '/var/lib/heketi/mounts/vg_1517317e9a1fc96c830c9493c49833f9': Directory not empty >] >[cmdexec] ERROR 2018/09/17 08:11:04 /src/github.com/heketi/heketi/executors/cmdexec/device.go:102: Error while removing the VG directory >[heketi] INFO 2018/09/17 08:11:05 Deleted node [1517317e9a1fc96c830c9493c49833f9] >[asynchttp] INFO 2018/09/17 08:11:05 asynchttp.go:292: Completed job 0984613fd7fc475ff5b48e5eddf9cd67 in 543.318528ms >[negroni] Started GET /queue/0984613fd7fc475ff5b48e5eddf9cd67 >[negroni] Completed 204 No Content in 203.036µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 277.945µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 451.604µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.145471ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.180234ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.007675ms >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:11:17 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 151.275538ms >[asynchttp] INFO 2018/09/17 08:11:17 asynchttp.go:288: Started job aecba3ae5824d91cb33e789db8aaef1e >[negroni] Started GET /queue/aecba3ae5824d91cb33e789db8aaef1e >[negroni] Completed 200 OK in 130.888µs >[kubeexec] ERROR 2018/09/17 08:11:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:11:18 asynchttp.go:292: Completed job aecba3ae5824d91cb33e789db8aaef1e in 548.93215ms >[negroni] Started GET /queue/aecba3ae5824d91cb33e789db8aaef1e >[negroni] Completed 500 Internal Server Error in 212.562µs >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:11:27 Adding device /dev/sdf to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 203.704589ms >[asynchttp] INFO 2018/09/17 08:11:27 asynchttp.go:288: Started job f90b6f6b5bae3a74e612f0f2d63a8b65 >[negroni] Started GET /queue/f90b6f6b5bae3a74e612f0f2d63a8b65 >[negroni] Completed 200 OK in 209.433µs >[kubeexec] ERROR 2018/09/17 08:11:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdf exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:11:28 asynchttp.go:292: Completed job f90b6f6b5bae3a74e612f0f2d63a8b65 in 288.038456ms >[negroni] Started GET /queue/f90b6f6b5bae3a74e612f0f2d63a8b65 >[negroni] Completed 500 Internal Server Error in 147.886µs >[heketi] INFO 2018/09/17 08:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:25:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:27:21 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 159.378001ms >[asynchttp] INFO 2018/09/17 08:27:21 asynchttp.go:288: Started job e1e53e65c603817c606a64fe6f63a14d >[negroni] Started GET /queue/e1e53e65c603817c606a64fe6f63a14d >[negroni] Completed 200 OK in 117.636µs >[kubeexec] ERROR 2018/09/17 08:27:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:27:22 asynchttp.go:292: Completed job e1e53e65c603817c606a64fe6f63a14d in 667.399638ms >[negroni] Started GET /queue/e1e53e65c603817c606a64fe6f63a14d >[negroni] Completed 500 Internal Server Error in 213.609µs >[heketi] INFO 2018/09/17 08:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:27:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:28:11 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 197.371465ms >[asynchttp] INFO 2018/09/17 08:28:11 asynchttp.go:288: Started job fa59f08a67ceb2fd502ceb5e151a29c3 >[negroni] Started GET /queue/fa59f08a67ceb2fd502ceb5e151a29c3 >[negroni] Completed 200 OK in 157.547µs >[kubeexec] ERROR 2018/09/17 08:28:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:28:12 asynchttp.go:292: Completed job fa59f08a67ceb2fd502ceb5e151a29c3 in 729.035419ms >[negroni] Started GET /queue/fa59f08a67ceb2fd502ceb5e151a29c3 >[negroni] Completed 500 Internal Server Error in 215.702µs >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 08:28:23 Adding device /dev/sdf to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 163.646365ms >[asynchttp] INFO 2018/09/17 08:28:23 asynchttp.go:288: Started job 3d398c815ff87a15483a580095f721c8 >[negroni] Started GET /queue/3d398c815ff87a15483a580095f721c8 >[negroni] Completed 200 OK in 191.216µs >[kubeexec] ERROR 2018/09/17 08:28:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdf exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 08:28:23 asynchttp.go:292: Completed job 3d398c815ff87a15483a580095f721c8 in 292.775851ms >[negroni] Started GET /queue/3d398c815ff87a15483a580095f721c8 >[negroni] Completed 500 Internal Server Error in 221.539µs >[heketi] INFO 2018/09/17 08:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 08:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 08:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 08:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 08:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 08:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 08:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 08:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 08:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 08:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 08:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 08:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:01:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 09:03:48 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 151.998873ms >[asynchttp] INFO 2018/09/17 09:03:48 asynchttp.go:288: Started job 8faf41623b606e613cc6fe0a06e9976c >[negroni] Started GET /queue/8faf41623b606e613cc6fe0a06e9976c >[negroni] Completed 200 OK in 128.735µs >[kubeexec] ERROR 2018/09/17 09:03:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 09:03:49 asynchttp.go:292: Completed job 8faf41623b606e613cc6fe0a06e9976c in 535.56982ms >[negroni] Started GET /queue/8faf41623b606e613cc6fe0a06e9976c >[negroni] Completed 500 Internal Server Error in 221.306µs >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 09:03:56 Adding device /dev/sdf to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 164.12344ms >[asynchttp] INFO 2018/09/17 09:03:56 asynchttp.go:288: Started job a750468701902f9c2273f9c9c4a094ae >[negroni] Started GET /queue/a750468701902f9c2273f9c9c4a094ae >[negroni] Completed 200 OK in 230.133µs >[kubeexec] ERROR 2018/09/17 09:03:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdf exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 09:03:56 asynchttp.go:292: Completed job a750468701902f9c2273f9c9c4a094ae in 288.37255ms >[negroni] Started GET /queue/a750468701902f9c2273f9c9c4a094ae >[negroni] Completed 500 Internal Server Error in 233.522µs >[heketi] INFO 2018/09/17 09:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:17 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[heketi] INFO 2018/09/17 09:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=false >[heketi] INFO 2018/09/17 09:21:59 Cleaned 0 nodes from health cache >[kubeexec] ERROR 2018/09/17 09:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [systemctl status glusterd] on glusterfs-cns-6hrtj: Err[command terminated with exit code 3]: Stdout [â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: activating (start) since Mon 2018-09-17 09:21:58 UTC; 1s ago > Main PID: 458 (code=exited, status=15); : 2782 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1359 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c > ââ2782 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ2880 /usr/bin/python /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py -c /var/lib/glusterd/geo-replication/gsyncd_template.conf --config-set-rx socketdir /var/run/gluster . . >]: Stderr [] >[cmdexec] ERROR 2018/09/17 09:21:59 /src/github.com/heketi/heketi/executors/cmdexec/peer.go:76: >[heketi] INFO 2018/09/17 09:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1min 58s ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3min 58s ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 5min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 7min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 9min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 11min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 13min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 15min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 17min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 19min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 21min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 23min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 25min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 27min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 29min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 31min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 33min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 35min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 09:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 09:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 09:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 09:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 09:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 09:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 09:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 09:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 09:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 37min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 09:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 09:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 39min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 41min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 43min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 45min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 47min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 49min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 51min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 53min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 55min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 57min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 59min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 1min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 3min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:28:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 5min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:28:00 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:28:00 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 7min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 9min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 11min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 13min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 15min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 17min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 19min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 21min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 23min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 25min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 27min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 29min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 31min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 33min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 35min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 10:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 10:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 10:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 10:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 10:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 10:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 10:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 10:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 10:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 37min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 10:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 10:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 39min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 41min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 43min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 45min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 47min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 49min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 51min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 53min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 55min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 57min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 1h 59min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 4 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 4 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 1min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 3min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 5min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 7min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 9min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 11min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 13min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 15min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 17min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 19min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 21min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 23min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 25min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 27min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 29min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 31min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 33min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 35min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 11:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 11:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 11:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 11:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 11:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 11:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 11:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 11:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 11:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 37min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 11:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 11:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 39min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 41min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 43min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 45min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 47min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 49min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 51min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 53min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 55min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 57min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:19:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 2h 59min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 1min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 3min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 5min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 7min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:29:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:29:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 9min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 11min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 13min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 15min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 17min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 19min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 21min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 23min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 25min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 27min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 29min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 31min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 33min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 35min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:57:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 12:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 12:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 12:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 12:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 12:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 12:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 12:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 12:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 12:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 37min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 12:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 12:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 39min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 41min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 43min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:05:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:07:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:07:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:07:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:07:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:07:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 45min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:07:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:07:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:09:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:09:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:09:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:09:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:09:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 47min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:09:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:09:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:11:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:11:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:11:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:11:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 49min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:11:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:11:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:13:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:13:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:13:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:13:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:13:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 51min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:13:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:13:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:15:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:15:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:15:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:15:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:15:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 53min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:15:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:15:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:17:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:17:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:17:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:17:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:17:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 55min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:17:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:17:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:19:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:19:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:19:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:19:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:19:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 57min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:19:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:19:59 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 272.622µs >[negroni] Started GET /volumes/4ea6274ee9959436d0b302ede4103ca0 >[negroni] Completed 200 OK in 1.14709ms >[negroni] Started GET /volumes/ffede5b0a8ce9555bf24fcc762e02b2c >[negroni] Completed 200 OK in 1.058511ms >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 221.096µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 419.241µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.029533ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.146161ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.032809ms >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 13:20:45 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 158.178308ms >[asynchttp] INFO 2018/09/17 13:20:45 asynchttp.go:288: Started job afd32c2d8aca6d5a43a927661916ba6b >[negroni] Started GET /queue/afd32c2d8aca6d5a43a927661916ba6b >[negroni] Completed 200 OK in 96.832µs >[kubeexec] ERROR 2018/09/17 13:20:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 13:20:45 asynchttp.go:292: Completed job afd32c2d8aca6d5a43a927661916ba6b in 490.943249ms >[negroni] Started GET /queue/afd32c2d8aca6d5a43a927661916ba6b >[negroni] Completed 500 Internal Server Error in 206.313µs >[heketi] INFO 2018/09/17 13:21:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:21:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:21:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:21:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:21:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 3h 59min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:21:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:21:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:23:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:23:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:23:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:23:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:23:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 1min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:23:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:23:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:25:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:25:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:25:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:25:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:25:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 3min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:25:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:25:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:27:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:27:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:27:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:27:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:27:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 5min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:27:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:27:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:29:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:29:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:29:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:29:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:29:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:30:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 7min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:30:00 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:30:00 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:31:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:31:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:31:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:31:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:31:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 9min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:31:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:31:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:33:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:33:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:33:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:33:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:33:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 11min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:33:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:33:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:35:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:35:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:35:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:35:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:35:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 13min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:35:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:35:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:37:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:37:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:37:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:37:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:37:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 15min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:37:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:37:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:39:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:39:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:39:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:39:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:39:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 17min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:39:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:39:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:41:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:41:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:41:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:41:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 19min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:41:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:41:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:43:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:43:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:43:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:43:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:43:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 21min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:43:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:43:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:45:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:45:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:45:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:45:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 23min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:45:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:45:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:47:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:47:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:47:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:47:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:47:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 25min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:47:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:47:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:49:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:49:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:49:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:49:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:49:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 27min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:49:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:49:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:51:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:51:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:51:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:51:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 29min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:51:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:51:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:53:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:53:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:53:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:53:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:53:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 31min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:53:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:53:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:55:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:55:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:55:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:55:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:55:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 33min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:55:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:55:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 13:57:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:57:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:57:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:57:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:57:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 35min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:57:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:57:59 Cleaned 0 nodes from health cache >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 13:59:03 Adding device /dev/sdd to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 132.750441ms >[asynchttp] INFO 2018/09/17 13:59:03 asynchttp.go:288: Started job 0670b0cabba851cc0b68544b64be06a8 >[negroni] Started GET /queue/0670b0cabba851cc0b68544b64be06a8 >[negroni] Completed 200 OK in 231.159µs >[kubeexec] ERROR 2018/09/17 13:59:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdd'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdd exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 13:59:04 asynchttp.go:292: Completed job 0670b0cabba851cc0b68544b64be06a8 in 526.49319ms >[negroni] Started GET /queue/0670b0cabba851cc0b68544b64be06a8 >[negroni] Completed 500 Internal Server Error in 155.26µs >[negroni] Started POST /devices >[heketi] INFO 2018/09/17 13:59:19 Adding device /dev/sdf to node 8966bb38131b92389d55e09862ca9cce >[negroni] Completed 202 Accepted in 135.81739ms >[asynchttp] INFO 2018/09/17 13:59:19 asynchttp.go:288: Started job eff8bbf6a3e7683f1ad96e7c418a02d9 >[negroni] Started GET /queue/eff8bbf6a3e7683f1ad96e7c418a02d9 >[negroni] Completed 200 OK in 218.362µs >[kubeexec] ERROR 2018/09/17 13:59:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [pvcreate --metadatasize=128M --dataalignment=256K '/dev/sdf'] on glusterfs-cns-6hrtj: Err[command terminated with exit code 5]: Stdout []: Stderr [ Can't open /dev/sdf exclusively. Mounted filesystem? >] >[asynchttp] INFO 2018/09/17 13:59:19 asynchttp.go:292: Completed job eff8bbf6a3e7683f1ad96e7c418a02d9 in 394.777278ms >[negroni] Started GET /queue/eff8bbf6a3e7683f1ad96e7c418a02d9 >[negroni] Completed 500 Internal Server Error in 216.643µs >[negroni] Started GET /clusters >[negroni] Completed 200 OK in 226.212µs >[negroni] Started GET /clusters/066e95e71ca3fcfff378e94acdff7f34 >[negroni] Completed 200 OK in 470.391µs >[negroni] Started GET /nodes/216efa66fc7fb3f0516e3a8d0b5aed20 >[negroni] Completed 200 OK in 1.262401ms >[negroni] Started GET /nodes/4fe4613a9dd872b70a79563049a011c0 >[negroni] Completed 200 OK in 1.207094ms >[negroni] Started GET /nodes/8966bb38131b92389d55e09862ca9cce >[negroni] Completed 200 OK in 1.005922ms >[heketi] INFO 2018/09/17 13:59:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 13:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 13:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 13:59:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 13:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 13:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 13:59:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 13:59:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 13:59:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 37min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 13:59:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 13:59:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 14:01:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 14:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 14:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 14:01:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 14:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 14:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 14:01:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 14:01:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 14:01:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 39min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 14:01:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 14:01:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 14:03:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 14:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 14:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 14:03:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 14:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 14:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 14:03:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 14:03:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 14:03:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 41min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 14:03:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 14:03:59 Cleaned 0 nodes from health cache >[heketi] INFO 2018/09/17 14:05:59 Starting Node Health Status refresh >[cmdexec] INFO 2018/09/17 14:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-0 >[kubeexec] DEBUG 2018/09/17 14:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-0 Pod: glusterfs-cns-rkphq Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:28 UTC; 5 days ago > Process: 457 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 458 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd65d7_b67e_11e8_a796_005056a5140f.slice/docker-907b9f6f683768601900c22bb7f1c146fa79467fc9c5e801327b075342310923.scope/system.slice/glusterd.service > ââ 458 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 836 /usr/sbin/glusterfsd -s 10.70.47.41 --volfile-id heketidbstorage.10.70.47.41.var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.41-var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.pid -S /var/run/gluster/8e2ac613cfeeec773fd878d4b15cd61b.socket --brick-name /var/lib/heketi/mounts/vg_e9fa7f608e4dacb7d8477ced702b22dd/brick_862dfad9e7e06fde3aa0972bf89b9bdf/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e9fa7f608e4dacb7d8477ced702b22dd-brick_862dfad9e7e06fde3aa0972bf89b9bdf-brick.log --xlator-option *-posix.glusterd-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1167 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/39d2be9067d3ce3faaccb6a603119b96.socket --xlator-option *replicate*.node-uuid=15f88a3a-4bd3-4343-a3da-4a4b88efd281 >[heketi] INFO 2018/09/17 14:05:59 Periodic health check status: node 216efa66fc7fb3f0516e3a8d0b5aed20 up=true >[cmdexec] INFO 2018/09/17 14:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-2 >[kubeexec] DEBUG 2018/09/17 14:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-2 Pod: glusterfs-cns-cq2sc Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-09-12 11:25:39 UTC; 5 days ago > Process: 451 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 452 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420d2d34_b67e_11e8_a796_005056a5140f.slice/docker-fb6b374d5541fc34ba001c1f126a8211d5ea80246e76e26e22f7ab49285c2355.scope/system.slice/glusterd.service > ââ 452 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 932 /usr/sbin/glusterfsd -s 10.70.46.57 --volfile-id heketidbstorage.10.70.46.57.var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.57-var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.pid -S /var/run/gluster/4004bda96f988d001b61426285008f17.socket --brick-name /var/lib/heketi/mounts/vg_b08bde372ba7b620befd7edef290a896/brick_41234f2611768363d4dad45fc66c340f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b08bde372ba7b620befd7edef290a896-brick_41234f2611768363d4dad45fc66c340f-brick.log --xlator-option *-posix.glusterd-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ1146 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/f4524abcbc15840087dffd9be42ebca2.socket --xlator-option *replicate*.node-uuid=fc918f48-75c7-435c-9ed2-4494074a7b92 >[heketi] INFO 2018/09/17 14:05:59 Periodic health check status: node 4fe4613a9dd872b70a79563049a011c0 up=true >[cmdexec] INFO 2018/09/17 14:05:59 Check Glusterd service status in node vp-ansible-v310-ga-app-cns-1 >[kubeexec] DEBUG 2018/09/17 14:05:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: vp-ansible-v310-ga-app-cns-1 Pod: glusterfs-cns-6hrtj Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Mon 2018-09-17 09:22:01 UTC; 4h 43min ago > Process: 2782 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 2783 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f6d41d_b67e_11e8_a796_005056a5140f.slice/docker-a8375d1603e2cd7da0983e5d5ab3892ae209a51aa8258a3fd2879d6654abe539.scope/system.slice/glusterd.service > ââ 837 /usr/sbin/glusterfsd -s 10.70.46.45 --volfile-id heketidbstorage.10.70.46.45.var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.45-var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.pid -S /var/run/gluster/754325ddf78caa52da0e937a5bc21c89.socket --brick-name /var/lib/heketi/mounts/vg_68c9ef3f5a2e31d2976565f9f187a6cf/brick_e04cc4729e00e4ec21aa1b028651d599/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_68c9ef3f5a2e31d2976565f9f187a6cf-brick_e04cc4729e00e4ec21aa1b028651d599-brick.log --xlator-option *-posix.glusterd-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ2783 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ3003 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c014d700bb91466d2d2ccb47a0910571.socket --xlator-option *replicate*.node-uuid=d39da99a-cbb9-44cb-9b3b-ae3cbbf1190c >[heketi] INFO 2018/09/17 14:05:59 Periodic health check status: node 8966bb38131b92389d55e09862ca9cce up=true >[heketi] INFO 2018/09/17 14:05:59 Cleaned 0 nodes from health cache
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Raw
Actions:
View
Attachments on
bug 1629889
:
1484065
| 1484066