Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1472656 Details for
Bug 1584123
In a CNS setup, heketi-cli info on a block-hosting volume displays "Free Size" greater than its actual size
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
heketi logs from upgraded setup
heketi_log_latest (text/plain), 171.51 KB, created by
Neha Berry
on 2018-08-02 11:19:35 UTC
(
hide
)
Description:
heketi logs from upgraded setup
Filename:
MIME Type:
Creator:
Neha Berry
Created:
2018-08-02 11:19:35 UTC
Size:
171.51 KB
patch
obsolete
>Heketi 7.0.0 >[heketi] INFO 2018/08/02 10:52:36 Loaded kubernetes executor >[heketi] INFO 2018/08/02 10:52:36 Please refer to the Heketi troubleshooting documentation for more information on how to resolve this issue. >[heketi] WARNING 2018/08/02 10:52:36 Ignoring stale pending operations.Server will be running with incomplete/inconsistent state in DB. >[heketi] INFO 2018/08/02 10:52:36 Block: Auto Create Block Hosting Volume set to true >[heketi] INFO 2018/08/02 10:52:36 Block: New Block Hosting Volume size 100 GB >[heketi] INFO 2018/08/02 10:52:36 GlusterFS Application Loaded >[heketi] ERROR 2018/08/02 10:52:36 /src/github.com/heketi/heketi/apps/glusterfs/app.go:176: Heketi was terminated while performing one or more operations. Server may refuse to start as long as pending operations are present in the db. >[heketi] INFO 2018/08/02 10:52:36 Started Node Health Cache Monitor >Authorization loaded >Listening on port 8080 >[heketi] INFO 2018/08/02 10:52:46 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 10:52:46 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:52:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:52:46 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 10:52:46 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:52:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:52:46 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 10:52:46 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:52:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:52:46 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 10:52:46 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 10:54:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 10:54:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:54:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 10:54:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:54:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 10:54:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:54:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 10:54:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 10:56:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 10:56:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:56:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 10:56:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:56:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 10:56:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:56:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:56:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 10:56:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 10:58:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 10:58:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:58:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 10:58:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:58:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 10:58:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 10:58:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 10:58:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 10:58:36 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 1.705831ms >[negroni] Started GET /volumes/33bed42d0e5eec32a53dd5b8ff35b509 >[negroni] Completed 200 OK in 1.360541ms >[negroni] Started GET /volumes/4e89f29a2ddc1d38e49de181194a3d21 >[negroni] Completed 200 OK in 653.514µs >[negroni] Started GET /volumes/c0706ceff6d4e1a67e406b1834b512e2 >[negroni] Completed 200 OK in 531.858µs >[negroni] Started GET /volumes/cf7456fa1c6c3e93725ed19faf42ce79 >[negroni] Completed 200 OK in 676.05µs >[negroni] Started GET /volumes/eab0ef054d1142e1b6a41527760d54aa >[negroni] Completed 200 OK in 668.326µs >[negroni] Started GET /blockvolumes >[negroni] Completed 200 OK in 2.485166ms >[negroni] Started GET /blockvolumes/0ae9027b9f70780f0316b1b8448dfe7d >[negroni] Completed 200 OK in 313.951µs >[negroni] Started GET /blockvolumes/0f02c63dc9454a107d5c533cbf846c60 >[negroni] Completed 200 OK in 217.41µs >[negroni] Started GET /blockvolumes/1559a190e895726b917039041d6dd6e9 >[negroni] Completed 200 OK in 184.926µs >[negroni] Started GET /blockvolumes/18a6f7397c14d5600b6b379b04de9055 >[negroni] Completed 200 OK in 193.255µs >[negroni] Started GET /blockvolumes/199b92ecde07beb039b2d12781631103 >[negroni] Completed 200 OK in 185.094µs >[negroni] Started GET /blockvolumes/2c9c161faca355e57f6bc00b45f9c316 >[negroni] Completed 200 OK in 205.581µs >[negroni] Started GET /blockvolumes/2f2dc8a9881b45374faa99e3b8194c66 >[negroni] Completed 200 OK in 226.006µs >[negroni] Started GET /blockvolumes/363c37a33e2f43148fab89f360ac4b76 >[negroni] Completed 200 OK in 186.591µs >[negroni] Started GET /blockvolumes/3f361422d5a4e3b7544dfde919798ef5 >[negroni] Completed 200 OK in 231.413µs >[negroni] Started GET /blockvolumes/43e118d399dba856b33ea5db93d7a621 >[negroni] Completed 200 OK in 207.817µs >[negroni] Started GET /blockvolumes/4a5e22950c09d8fdf7475aac6d06a574 >[negroni] Completed 200 OK in 204.24µs >[negroni] Started GET /blockvolumes/4ae2e3acaffc65dc5d1e761901ba5881 >[negroni] Completed 200 OK in 209.631µs >[negroni] Started GET /blockvolumes/505a7d71d2435bd1a574e0d7d4b8499b >[negroni] Completed 200 OK in 251.578µs >[negroni] Started GET /blockvolumes/55404027415b32b330376b861efee128 >[negroni] Completed 200 OK in 245.819µs >[negroni] Started GET /blockvolumes/5a729b8e8a04817b34dcec4c4a3ae3a9 >[negroni] Completed 200 OK in 237.019µs >[negroni] Started GET /blockvolumes/5fc3cb2a98f73dfb902b0ba144e4ea4a >[negroni] Completed 200 OK in 235.513µs >[negroni] Started GET /blockvolumes/60341462c4e96beca5c7aa597529e40f >[negroni] Completed 200 OK in 194.267µs >[negroni] Started GET /blockvolumes/62fc1df4c7edcadc583254e95d141a6e >[negroni] Completed 200 OK in 178.939µs >[negroni] Started GET /blockvolumes/69b3a4040de5d847fc4157c5149ce0c8 >[negroni] Completed 200 OK in 190.115µs >[negroni] Started GET /blockvolumes/71862bf3ac18087d28e67dc1ab36db65 >[negroni] Completed 200 OK in 187.048µs >[negroni] Started GET /blockvolumes/7b6c1f877e0ffd419e839f45aedbaaf3 >[negroni] Completed 200 OK in 186.588µs >[negroni] Started GET /blockvolumes/82217a727c58d622ccdb0d654a0ee41f >[negroni] Completed 200 OK in 183.163µs >[negroni] Started GET /blockvolumes/844e171e3b1382fce1fcbf2d5450da51 >[negroni] Completed 200 OK in 183.998µs >[negroni] Started GET /blockvolumes/89f6e9cf7ab1a7cab916bdc5ab89aa6e >[negroni] Completed 200 OK in 208.032µs >[negroni] Started GET /blockvolumes/8bec3690e3b951703f75b91361667e19 >[negroni] Completed 200 OK in 342.687µs >[negroni] Started GET /blockvolumes/937df2d124e891abb3862cc83b27aabd >[negroni] Completed 200 OK in 258.329µs >[negroni] Started GET /blockvolumes/a69a784c9104ffbcdec2bf9d8f3a141d >[negroni] Completed 200 OK in 272.179µs >[negroni] Started GET /blockvolumes/a8f61806fc9bafa10c404db7b9ef8911 >[negroni] Completed 200 OK in 189.875µs >[negroni] Started GET /blockvolumes/ab96e32c6902093190f1317c24aab0bf >[negroni] Completed 200 OK in 190.823µs >[negroni] Started GET /blockvolumes/aee807e76f7063f7e38054168b80d86f >[negroni] Completed 200 OK in 185.86µs >[negroni] Started GET /blockvolumes/b931170c9a5f7fe802e1d82eb45525e8 >[negroni] Completed 200 OK in 199.192µs >[negroni] Started GET /blockvolumes/b9c61ce96f064de6aad3ccc7cda51337 >[negroni] Completed 200 OK in 180.384µs >[negroni] Started GET /blockvolumes/bbd544539eb6ab50b174f04fc2090f40 >[negroni] Completed 200 OK in 207.942µs >[negroni] Started GET /blockvolumes/bd35030c86a17b309a3e5059636c7363 >[negroni] Completed 200 OK in 236.737µs >[negroni] Started GET /blockvolumes/c4243252bd429b4b0cd4f87a5d984b5a >[negroni] Completed 200 OK in 194.473µs >[negroni] Started GET /blockvolumes/c5612338684a8406503b05b220c77e93 >[negroni] Completed 200 OK in 333.799µs >[negroni] Started GET /blockvolumes/d516ce72aa3dcdcd10c2405aebebae05 >[negroni] Completed 200 OK in 246.682µs >[negroni] Started GET /blockvolumes/d65e89b906a709cc90674b947427dbe4 >[negroni] Completed 200 OK in 230.39µs >[negroni] Started GET /blockvolumes/e0431473be783e8d7f18827dd8b06080 >[negroni] Completed 200 OK in 190.155µs >[negroni] Started GET /blockvolumes/e3f55620749b3c7bcec4dd0e670250f4 >[negroni] Completed 200 OK in 212.621µs >[negroni] Started GET /blockvolumes/e66bc79263c12cb0a993710ff6641d05 >[negroni] Completed 200 OK in 181.54µs >[negroni] Started GET /blockvolumes/ed44482204cc49aec2e37b0cc2bc67a0 >[negroni] Completed 200 OK in 181.007µs >[negroni] Started GET /blockvolumes/f75a39d89e973cd567395fbe7e460e03 >[negroni] Completed 200 OK in 195.549µs >[negroni] Started GET /blockvolumes/f9dfff25d32619b201ac461150bbb6b5 >[negroni] Completed 200 OK in 187.613µs >[negroni] Started GET /volumes/33bed42d0e5eec32a53dd5b8ff35b509 >[negroni] Completed 200 OK in 726.718µs >[heketi] INFO 2018/08/02 11:00:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:00:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:00:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:00:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:00:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:00:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:00:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:00:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:00:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:00:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:00:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:02:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:02:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:02:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:02:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:02:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:02:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:02:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:02:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:02:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:02:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:02:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:04:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:04:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:04:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:04:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:04:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:04:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:04:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:04:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:04:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:06:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:06:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:06:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:06:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:06:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:06:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:06:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:06:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:06:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:08:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:08:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:08:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:08:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:08:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:08:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:08:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:08:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:10:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:10:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:10:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:10:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:10:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:10:37 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:10:37 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:10:37 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:10:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:12:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:12:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:12:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:12:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:12:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:12:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:12:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:12:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:12:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:12:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:12:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:14:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:14:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:14:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:14:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:14:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:14:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:14:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:14:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:14:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:14:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:14:36 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/02 11:16:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/02 11:16:36 Check Glusterd service status in node dhcp42-84.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:16:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-84.lab.eng.blr.redhat.com Pod: glusterfs-storage-lzmlr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 09:04:22 UTC; 2 days ago > Process: 180 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 182 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ecea4b_94a0_11e8_9bce_005056a52b66.slice/docker-250112bf09ea2d0da3193509aea7485e820939e63df09b643ce3aff00af3fc6a.scope/system.slice/glusterd.service > ââ 182 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 430 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id heketidbstorage.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.pid -S /var/run/gluster/baa353f83e4be8c878f41fc3d35a790d.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_ece938d386f0598ea4325574773a311f/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_ece938d386f0598ea4325574773a311f-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 > ââ 438 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.pid -S /var/run/gluster/df0f7271205fc4d9c3c7aa73094a5113.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_63d9427010c196085cde0783f9aacb8c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_63d9427010c196085cde0783f9aacb8c-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49155 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49155 > ââ54542 /usr/sbin/glusterfsd -s 10.70.42.84 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.84.var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.84-var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.pid -S /var/run/gluster/f1ed79fcf29d6b19be273457d1ab371a.socket --brick-name /var/lib/heketi/mounts/vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b/brick_aa0fa4ff446c692abc3b862b050d3e2e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_ae9f9ce2bedb00fd27e05ddc9ca76b8b-brick_aa0fa4ff446c692abc3b862b050d3e2e-brick.log --xlator-option *-posix.glusterd-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f --brick-port 49152 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49152 > ââ54562 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/51860da1bfdedff68e47a695151f9781.socket --xlator-option *replicate*.node-uuid=34748a0f-b2f4-4a85-a083-60b486b2c53f > >Jul 31 09:04:20 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 09:04:22 dhcp42-84.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:16:36 Periodic health check status: node 48c8e47a1cb9b2a7bd310caadda60d7d up=true >[cmdexec] INFO 2018/08/02 11:16:36 Check Glusterd service status in node dhcp41-217.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:16:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp41-217.lab.eng.blr.redhat.com Pod: glusterfs-storage-c9snw Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Tue 2018-07-31 10:08:31 UTC; 2 days ago > Process: 186 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 188 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac81c2a_94a9_11e8_9bce_005056a52b66.slice/docker-faef0c4620bae109ad302fbab4b6d5554564592bf5c73762ae6338ab14d79c57.scope/system.slice/glusterd.service > ââ 188 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 437 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id heketidbstorage.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick -p /var/run/gluster/vols/heketidbstorage/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.pid -S /var/run/gluster/d14ceb95cc601e56f9ee9f68197fb3bb.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_084b01adfc1752f866f49c67510bf783/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_084b01adfc1752f866f49c67510bf783-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 446 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.41.217.var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.41.217-var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.pid -S /var/run/gluster/d2b090733b5d320429a4df713d300f12.socket --brick-name /var/lib/heketi/mounts/vg_2039cb0938d6f1883ffc4a5aad2535e7/brick_36eace773eea199daf7775d3498bbe87/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_2039cb0938d6f1883ffc4a5aad2535e7-brick_36eace773eea199daf7775d3498bbe87-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ46058 /usr/sbin/glusterfsd -s 10.70.41.217 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.41.217.var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.41.217-var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.pid -S /var/run/gluster/b04a94510456cce916582c7f90bded46.socket --brick-name /var/lib/heketi/mounts/vg_893d06d8826c56927a7902344f57dab1/brick_d515c68b6ba917d81c7da5b6f91d486a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_893d06d8826c56927a7902344f57dab1-brick_d515c68b6ba917d81c7da5b6f91d486a-brick.log --xlator-option *-posix.glusterd-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ46078 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/30c00ad63f09b5c2be30b69001bed5d4.socket --xlator-option *replicate*.node-uuid=96c5c62a-acf2-43a0-9bfe-55f3f69757d6 > >Jul 31 10:08:28 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Jul 31 10:08:31 dhcp41-217.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:16:36 Periodic health check status: node 6343c9dbbd35e81055c4971061a4ee1c up=true >[cmdexec] INFO 2018/08/02 11:16:36 Check Glusterd service status in node dhcp42-223.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/02 11:16:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp42-223.lab.eng.blr.redhat.com Pod: glusterfs-storage-m4qpp Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Wed 2018-08-01 08:38:52 UTC; 1 day 2h ago > Process: 255 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 258 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80b22ea_9565_11e8_9bce_005056a52b66.slice/docker-c94c08658e6b873ab7bf1d72d6af6b2e37b191e66091d86a059db623c5906044.scope/system.slice/glusterd.service > ââ 258 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 506 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id heketidbstorage.10.70.42.223.var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick -p /var/run/gluster/vols/heketidbstorage/10.70.42.223-var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.pid -S /var/run/gluster/d061cc95fbd0c8cbc72e26deca104fc8.socket --brick-name /var/lib/heketi/mounts/vg_b0003ccff9831947b60cf6a29e2bdc0d/brick_511486b46faeb1da6aaf987982c3676a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_b0003ccff9831947b60cf6a29e2bdc0d-brick_511486b46faeb1da6aaf987982c3676a-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49152 --xlator-option heketidbstorage-server.listen-port=49152 > ââ 657 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_33bed42d0e5eec32a53dd5b8ff35b509.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick -p /var/run/gluster/vols/vol_33bed42d0e5eec32a53dd5b8ff35b509/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.pid -S /var/run/gluster/8408d3d7a43139b667615b913a9c2369.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_cd4984dcb2535f923b42dc777011c071/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_cd4984dcb2535f923b42dc777011c071-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49153 --xlator-option vol_33bed42d0e5eec32a53dd5b8ff35b509-server.listen-port=49153 > ââ7021 /usr/sbin/glusterfsd -s 10.70.42.223 --volfile-id vol_4e89f29a2ddc1d38e49de181194a3d21.10.70.42.223.var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick -p /var/run/gluster/vols/vol_4e89f29a2ddc1d38e49de181194a3d21/10.70.42.223-var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.pid -S /var/run/gluster/6b64ca29f20800cac4935382ee6bab9b.socket --brick-name /var/lib/heketi/mounts/vg_e6e29849050ffa560fd7456d4c6e6879/brick_a0598cefe4ac9ac2d8f7a5025457e03b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e6e29849050ffa560fd7456d4c6e6879-brick_a0598cefe4ac9ac2d8f7a5025457e03b-brick.log --xlator-option *-posix.glusterd-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f --brick-port 49154 --xlator-option vol_4e89f29a2ddc1d38e49de181194a3d21-server.listen-port=49154 > ââ7041 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/262b3a23b489a78a4d9b738ba23e553e.socket --xlator-option *replicate*.node-uuid=a4be6d6e-8a8e-4b1a-b1f3-5fe593440e6f > >Aug 01 08:38:49 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Starting GlusterFS, a clustered file-system server... >Aug 01 08:38:52 dhcp42-223.lab.eng.blr.redhat.com systemd[1]: Started GlusterFS, a clustered file-system server. >[heketi] INFO 2018/08/02 11:16:36 Periodic health check status: node 689a23c46ec24d0beb02b555af17dcb6 up=true >[heketi] INFO 2018/08/02 11:16:36 Cleaned 0 nodes from health cache
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1472656">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1584123
: 1472656