Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1476922 Details for
Bug 1619017
mismatch in number of block devices between heketi and gluster
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
heketi_logs
heketilogs_1619017 (text/plain), 6.19 MB, created by
krishnaram Karthick
on 2018-08-19 17:09:17 UTC
(
hide
)
Description:
heketi_logs
Filename:
MIME Type:
Creator:
krishnaram Karthick
Created:
2018-08-19 17:09:17 UTC
Size:
6.19 MB
patch
obsolete
>Heketi 7.0.0 >[heketi] INFO 2018/08/19 11:56:36 Loaded kubernetes executor >[heketi] INFO 2018/08/19 11:56:36 Block: Auto Create Block Hosting Volume set to true >[heketi] INFO 2018/08/19 11:56:36 Block: New Block Hosting Volume size 100 GB >[heketi] INFO 2018/08/19 11:56:36 GlusterFS Application Loaded >[heketi] INFO 2018/08/19 11:56:36 Started Node Health Cache Monitor >Authorization loaded >Listening on port 8080 >[heketi] INFO 2018/08/19 11:56:46 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 11:56:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:56:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36s ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 11:56:47 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 11:56:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:56:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 28s ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 11:56:47 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 11:56:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:56:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 28s ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 11:56:47 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 11:56:47 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 11:58:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 11:58:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2min 26s ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 11:58:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 11:58:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2min 18s ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 11:58:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 11:58:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 11:58:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2min 19s ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 11:58:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 11:58:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:00:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:00:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 4min 26s ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 12:00:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:00:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 4min 18s ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:00:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:00:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 4min 18s ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:00:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:00:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:02:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:02:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 12:02:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:02:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:02:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:02:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 6min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:02:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:02:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:04:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:04:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 12:04:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:04:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:04:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:04:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 8min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:04:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:04:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:06:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:06:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ676 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 > ââ694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 >[heketi] INFO 2018/08/19 12:06:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:06:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ460 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 > ââ495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:06:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:06:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 10min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ430 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 12:06:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:06:37 Cleaned 0 nodes from health cache >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:06:51 Allocating brick set #0 >[negroni] Completed 202 Accepted in 22.993064ms >[asynchttp] INFO 2018/08/19 12:06:51 asynchttp.go:288: Started job 3df763972047fd7d4dcd63b9fc8de66a >[heketi] INFO 2018/08/19 12:06:51 Started async operation: Create Volume >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 81.382µs >[heketi] INFO 2018/08/19 12:06:51 Creating brick 53edecfa26afca6c04e10aedf0ce3142 >[heketi] INFO 2018/08/19 12:06:51 Creating brick 86bf1fe4ee504b97331bdf87ded5f601 >[heketi] INFO 2018/08/19 12:06:51 Creating brick bfdeb7e9b9cf3f0dd0536389e1b47d4a >[kubeexec] DEBUG 2018/08/19 12:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142 >Result: >[kubeexec] DEBUG 2018/08/19 12:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601 >Result: >[kubeexec] DEBUG 2018/08/19 12:06:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_53edecfa26afca6c04e10aedf0ce3142 --virtualsize 1048576K --name brick_53edecfa26afca6c04e10aedf0ce3142 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_53edecfa26afca6c04e10aedf0ce3142" created. >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_bb9f1f0b51926ad9acd805d69d935e81 --virtualsize 1048576K --name brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a" created. >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_86bf1fe4ee504b97331bdf87ded5f601 --virtualsize 1048576K --name brick_86bf1fe4ee504b97331bdf87ded5f601 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_86bf1fe4ee504b97331bdf87ded5f601" created. >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_53edecfa26afca6c04e10aedf0ce3142 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_53edecfa26afca6c04e10aedf0ce3142 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_53edecfa26afca6c04e10aedf0ce3142 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 142.258µs >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601 >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2000 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_53edecfa26afca6c04e10aedf0ce3142 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142 >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2000 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2000 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:06:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142/brick >Result: >[cmdexec] INFO 2018/08/19 12:06:53 Creating volume vol_80ff41fdbb475c0fa055de08db90ab4f replica 3 >[kubeexec] DEBUG 2018/08/19 12:06:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_80ff41fdbb475c0fa055de08db90ab4f replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_bfdeb7e9b9cf3f0dd0536389e1b47d4a/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_53edecfa26afca6c04e10aedf0ce3142/brick >Result: volume create: vol_80ff41fdbb475c0fa055de08db90ab4f: success: please start the volume to access data >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 256.866µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 174.056µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 131.235µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 135.924µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 191.673µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 144.284µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 248.165µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 127.527µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 198.371µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 148.214µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 193.239µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 245.738µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 135.008µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 130.85µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 117.577µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 214.138µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 226.753µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 195.905µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 142.54µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 136.072µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 141.41µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 211.042µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 200.222µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 252.125µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 279.848µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 191.323µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 139.949µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 253.939µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 144.655µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 175.362µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 192.298µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 134.944µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 184.289µs >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 200 OK in 156.049µs >[kubeexec] DEBUG 2018/08/19 12:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_80ff41fdbb475c0fa055de08db90ab4f >Result: volume start: vol_80ff41fdbb475c0fa055de08db90ab4f: success >[heketi] INFO 2018/08/19 12:07:26 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:07:26 asynchttp.go:292: Completed job 3df763972047fd7d4dcd63b9fc8de66a in 35.086000672s >[negroni] Started GET /queue/3df763972047fd7d4dcd63b9fc8de66a >[negroni] Completed 303 See Other in 192.528µs >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 2.513515ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 224.262µs >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 1.189841ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 453.106µs >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 883.428µs >[heketi] INFO 2018/08/19 12:08:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:08:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:08:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:08:36 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:08:36 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:08:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:08:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 12min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:08:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:08:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 22.495013ms >[asynchttp] INFO 2018/08/19 12:10:13 asynchttp.go:288: Started job 79d5459cc74d2315c15617524b0f8996 >[heketi] INFO 2018/08/19 12:10:13 Started async operation: Create Block Volume >[negroni] Started GET /queue/79d5459cc74d2315c15617524b0f8996 >[negroni] Completed 200 OK in 109.186µs >[cmdexec] INFO 2018/08/19 12:10:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:10:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 13min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:10:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:10:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/79d5459cc74d2315c15617524b0f8996 >[negroni] Completed 200 OK in 154.281µs >[kubeexec] DEBUG 2018/08/19 12:10:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_d0b08f33621cbf7a7900873eee8dc0a9 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d4b84140-f9e4-4bf2-90dd-86ec5df93edc", "USERNAME": "d4b84140-f9e4-4bf2-90dd-86ec5df93edc", "PASSWORD": "e40a19c0-7510-4b8a-84ad-14c9d54062b6", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:10:15 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:10:15 asynchttp.go:292: Completed job 79d5459cc74d2315c15617524b0f8996 in 1.688373558s >[negroni] Started GET /queue/79d5459cc74d2315c15617524b0f8996 >[negroni] Completed 303 See Other in 179.653µs >[negroni] Started GET /blockvolumes/d0b08f33621cbf7a7900873eee8dc0a9 >[negroni] Completed 200 OK in 2.147571ms >[heketi] INFO 2018/08/19 12:10:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:10:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:10:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:10:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:10:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:10:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 14min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:10:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:10:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:12:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:12:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:12:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:12:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:12:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:12:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 16min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:12:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:12:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:14:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:14:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:14:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:14:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:14:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:14:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 18min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:14:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:14:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:16:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:16:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:16:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:16:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:16:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:16:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 20min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:16:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:16:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:18:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:18:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:18:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:18:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:18:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:18:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 22min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:18:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:18:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:20:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:20:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:20:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:20:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:20:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:20:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 24min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:20:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:20:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:22:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:22:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:22:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:22:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:22:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:22:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 26min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:22:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:22:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:24:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:24:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:24:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:24:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 28min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:24:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:24:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 28min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:24:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:24:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:26:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:26:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 30min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:26:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:26:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 30min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:26:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:26:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 30min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:26:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:26:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:28:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:28:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 32min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:28:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:28:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 32min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:28:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:28:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 32min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:28:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:28:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:30:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:30:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 34min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:30:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:30:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 34min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:30:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:30:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 34min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:30:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:30:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:32:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:32:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:32:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:32:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:32:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:32:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:32:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:32:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[asynchttp] INFO 2018/08/19 12:32:56 asynchttp.go:288: Started job 3c470f75a4e21c01ac14d26a1d9a6294 >[heketi] INFO 2018/08/19 12:32:56 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 22.941119ms >[negroni] Started GET /queue/3c470f75a4e21c01ac14d26a1d9a6294 >[negroni] Completed 200 OK in 107.64µs >[cmdexec] INFO 2018/08/19 12:32:56 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:32:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:32:56 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:32:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 11.678732ms >[asynchttp] INFO 2018/08/19 12:32:56 asynchttp.go:288: Started job 35392e86203404a20b4852034704780f >[heketi] INFO 2018/08/19 12:32:56 Started async operation: Create Block Volume >[negroni] Started GET /queue/35392e86203404a20b4852034704780f >[negroni] Completed 200 OK in 120.463µs >[cmdexec] INFO 2018/08/19 12:32:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3c470f75a4e21c01ac14d26a1d9a6294 >[negroni] Completed 200 OK in 163.545µs >[kubeexec] DEBUG 2018/08/19 12:32:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_ae15f47e58e1afb8f41532c2bc892676 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f70b21a8-02c2-4ea8-9e7e-7b6893675d3d", "USERNAME": "f70b21a8-02c2-4ea8-9e7e-7b6893675d3d", "PASSWORD": "0750ec12-3358-460c-a1ce-ba82635b62f9", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:32:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:32:57 asynchttp.go:292: Completed job 3c470f75a4e21c01ac14d26a1d9a6294 in 1.401581179s >[kubeexec] DEBUG 2018/08/19 12:32:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:32:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/35392e86203404a20b4852034704780f >[negroni] Completed 200 OK in 139.954µs >[kubeexec] DEBUG 2018/08/19 12:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:32:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:32:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:32:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3c470f75a4e21c01ac14d26a1d9a6294 >[negroni] Completed 303 See Other in 148.337µs >[negroni] Started GET /blockvolumes/ae15f47e58e1afb8f41532c2bc892676 >[negroni] Completed 200 OK in 580.37µs >[negroni] Started GET /queue/35392e86203404a20b4852034704780f >[negroni] Completed 200 OK in 156.576µs >[kubeexec] DEBUG 2018/08/19 12:32:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_734b361391d3cf14150a14f7a0a82632 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2a26ff61-4595-4474-bbd6-43933d66ece5", "USERNAME": "2a26ff61-4595-4474-bbd6-43933d66ece5", "PASSWORD": "1eaf1f97-7e5d-4469-b1e4-f70ce48e0215", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:32:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:32:59 asynchttp.go:292: Completed job 35392e86203404a20b4852034704780f in 2.063169623s >[negroni] Started GET /queue/35392e86203404a20b4852034704780f >[negroni] Completed 303 See Other in 215.916µs >[negroni] Started GET /blockvolumes/734b361391d3cf14150a14f7a0a82632 >[negroni] Completed 200 OK in 1.989469ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 10.13927ms >[asynchttp] INFO 2018/08/19 12:33:00 asynchttp.go:288: Started job 226142e8c09c6e9a18f6be6ec8589427 >[heketi] INFO 2018/08/19 12:33:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/226142e8c09c6e9a18f6be6ec8589427 >[negroni] Completed 200 OK in 109.258µs >[cmdexec] INFO 2018/08/19 12:33:00 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:00 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/226142e8c09c6e9a18f6be6ec8589427 >[negroni] Completed 200 OK in 120.428µs >[kubeexec] DEBUG 2018/08/19 12:33:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_296f9f8049fcd10e5ece2e901bc394e7 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d8d2f756-1b49-4ebe-a5d1-064e55b348b6", "USERNAME": "d8d2f756-1b49-4ebe-a5d1-064e55b348b6", "PASSWORD": "8df74a77-ee59-429c-8174-1cd1efe854ba", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:02 asynchttp.go:292: Completed job 226142e8c09c6e9a18f6be6ec8589427 in 1.428332663s >[negroni] Started POST /blockvolumes >[asynchttp] INFO 2018/08/19 12:33:02 asynchttp.go:288: Started job 58272de550a169f421fff6aa09d34ad9 >[heketi] INFO 2018/08/19 12:33:02 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 30.355566ms >[negroni] Started GET /queue/58272de550a169f421fff6aa09d34ad9 >[negroni] Completed 200 OK in 88.989µs >[cmdexec] INFO 2018/08/19 12:33:02 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/226142e8c09c6e9a18f6be6ec8589427 >[negroni] Completed 303 See Other in 143.513µs >[negroni] Started GET /blockvolumes/296f9f8049fcd10e5ece2e901bc394e7 >[negroni] Completed 200 OK in 677.01µs >[kubeexec] DEBUG 2018/08/19 12:33:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:02 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/58272de550a169f421fff6aa09d34ad9 >[negroni] Completed 200 OK in 148.276µs >[kubeexec] DEBUG 2018/08/19 12:33:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_803ddc4bd13e656f0cd960eaed4b9284 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ec44c49d-60cc-41c0-8f17-8870e78ba38f", "USERNAME": "ec44c49d-60cc-41c0-8f17-8870e78ba38f", "PASSWORD": "80ae3957-ca3e-483e-b867-b3738bb4ae68", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:03 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:03 asynchttp.go:292: Completed job 58272de550a169f421fff6aa09d34ad9 in 1.3375269s >[negroni] Started GET /queue/58272de550a169f421fff6aa09d34ad9 >[negroni] Completed 303 See Other in 166.973µs >[negroni] Started GET /blockvolumes/803ddc4bd13e656f0cd960eaed4b9284 >[negroni] Completed 200 OK in 1.623855ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 9.305228ms >[asynchttp] INFO 2018/08/19 12:33:06 asynchttp.go:288: Started job 724dcbd6d2585c161ca746d5e1062cdb >[heketi] INFO 2018/08/19 12:33:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/724dcbd6d2585c161ca746d5e1062cdb >[negroni] Completed 200 OK in 117.144µs >[cmdexec] INFO 2018/08/19 12:33:06 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/724dcbd6d2585c161ca746d5e1062cdb >[negroni] Completed 200 OK in 130.41µs >[kubeexec] DEBUG 2018/08/19 12:33:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c6fc4619ea4b826924e492375fd1aa98 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6e9dbd9d-d94d-4082-ba0a-a739a71df27c", "USERNAME": "6e9dbd9d-d94d-4082-ba0a-a739a71df27c", "PASSWORD": "2abf1129-9082-4af4-b933-08940f3f5feb", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:07 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:07 asynchttp.go:292: Completed job 724dcbd6d2585c161ca746d5e1062cdb in 1.432407569s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 9.890967ms >[asynchttp] INFO 2018/08/19 12:33:08 asynchttp.go:288: Started job 9a26dd1be576a2b553e6f79d7a04aa1f >[heketi] INFO 2018/08/19 12:33:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/9a26dd1be576a2b553e6f79d7a04aa1f >[negroni] Completed 200 OK in 91.126µs >[cmdexec] INFO 2018/08/19 12:33:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/724dcbd6d2585c161ca746d5e1062cdb >[negroni] Completed 303 See Other in 193.654µs >[negroni] Started GET /blockvolumes/c6fc4619ea4b826924e492375fd1aa98 >[negroni] Completed 200 OK in 617.945µs >[kubeexec] DEBUG 2018/08/19 12:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9a26dd1be576a2b553e6f79d7a04aa1f >[negroni] Completed 200 OK in 144.335µs >[kubeexec] DEBUG 2018/08/19 12:33:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1d0c5f8778c4fb0c4b444c284f300526 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ddbca8b3-cf32-4796-a604-34d7171d4fe6", "USERNAME": "ddbca8b3-cf32-4796-a604-34d7171d4fe6", "PASSWORD": "231fbc54-1b87-4237-8d15-28ca3cfb4fca", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:09 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:09 asynchttp.go:292: Completed job 9a26dd1be576a2b553e6f79d7a04aa1f in 1.21826458s >[negroni] Started GET /queue/9a26dd1be576a2b553e6f79d7a04aa1f >[negroni] Completed 303 See Other in 135.072µs >[negroni] Started GET /blockvolumes/1d0c5f8778c4fb0c4b444c284f300526 >[negroni] Completed 200 OK in 1.576048ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 56.424798ms >[asynchttp] INFO 2018/08/19 12:33:12 asynchttp.go:288: Started job cbcc85925051da8df8d1c33ba22f7a5d >[heketi] INFO 2018/08/19 12:33:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/cbcc85925051da8df8d1c33ba22f7a5d >[negroni] Completed 200 OK in 104.737µs >[cmdexec] INFO 2018/08/19 12:33:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:12 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:12 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cbcc85925051da8df8d1c33ba22f7a5d >[negroni] Completed 200 OK in 141.885µs >[kubeexec] DEBUG 2018/08/19 12:33:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c9a3c0de2187199b38a6eebf73d6b19a ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ac38b268-8960-40e6-a6de-c447fb556224", "USERNAME": "ac38b268-8960-40e6-a6de-c447fb556224", "PASSWORD": "ec4f3ba0-5606-48af-baa6-dc686ff48bc7", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:13 asynchttp.go:292: Completed job cbcc85925051da8df8d1c33ba22f7a5d in 1.397764624s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 12.954006ms >[asynchttp] INFO 2018/08/19 12:33:13 asynchttp.go:288: Started job edb4c723df439eb77e5e407f196daa33 >[heketi] INFO 2018/08/19 12:33:13 Started async operation: Create Block Volume >[negroni] Started GET /queue/edb4c723df439eb77e5e407f196daa33 >[negroni] Completed 200 OK in 143.385µs >[cmdexec] INFO 2018/08/19 12:33:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/cbcc85925051da8df8d1c33ba22f7a5d >[negroni] Completed 303 See Other in 125.551µs >[negroni] Started GET /blockvolumes/c9a3c0de2187199b38a6eebf73d6b19a >[negroni] Completed 200 OK in 1.085079ms >[kubeexec] DEBUG 2018/08/19 12:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/edb4c723df439eb77e5e407f196daa33 >[negroni] Completed 200 OK in 131.244µs >[kubeexec] DEBUG 2018/08/19 12:33:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_a50ecea6fbf8f8e45e581b261726a0c8 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:65210d55-1694-46df-8ef3-5c1482d6917c", "USERNAME": "65210d55-1694-46df-8ef3-5c1482d6917c", "PASSWORD": "b6a034c4-87a2-46cd-9c0d-08a4e9f36149", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:15 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:15 asynchttp.go:292: Completed job edb4c723df439eb77e5e407f196daa33 in 1.395008465s >[negroni] Started GET /queue/edb4c723df439eb77e5e407f196daa33 >[negroni] Completed 303 See Other in 158.312µs >[negroni] Started GET /blockvolumes/a50ecea6fbf8f8e45e581b261726a0c8 >[negroni] Completed 200 OK in 1.548402ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 10.960768ms >[asynchttp] INFO 2018/08/19 12:33:17 asynchttp.go:288: Started job 63d3ff9e8bab894df87a4b6e302fa0d8 >[heketi] INFO 2018/08/19 12:33:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/63d3ff9e8bab894df87a4b6e302fa0d8 >[negroni] Completed 200 OK in 211.498µs >[cmdexec] INFO 2018/08/19 12:33:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 36min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/63d3ff9e8bab894df87a4b6e302fa0d8 >[negroni] Completed 200 OK in 144.531µs >[kubeexec] DEBUG 2018/08/19 12:33:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_72541f809a1332bc00dd6123976a2e61 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a985a95f-1db0-4602-95c1-797788243171", "USERNAME": "a985a95f-1db0-4602-95c1-797788243171", "PASSWORD": "950f2878-fa6a-4e83-ae45-fa3aadfdfb0c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:18 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:18 asynchttp.go:292: Completed job 63d3ff9e8bab894df87a4b6e302fa0d8 in 1.309309072s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 10.572068ms >[asynchttp] INFO 2018/08/19 12:33:19 asynchttp.go:288: Started job 0b61b57ea0654a8b133a1c4c7f601b44 >[heketi] INFO 2018/08/19 12:33:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/0b61b57ea0654a8b133a1c4c7f601b44 >[negroni] Completed 200 OK in 83.974µs >[cmdexec] INFO 2018/08/19 12:33:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 37min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:33:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:33:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/63d3ff9e8bab894df87a4b6e302fa0d8 >[negroni] Completed 303 See Other in 217.883µs >[negroni] Started GET /blockvolumes/72541f809a1332bc00dd6123976a2e61 >[negroni] Completed 200 OK in 1.076059ms >[kubeexec] DEBUG 2018/08/19 12:33:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 37min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:33:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:33:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0b61b57ea0654a8b133a1c4c7f601b44 >[negroni] Completed 200 OK in 190.784µs >[kubeexec] DEBUG 2018/08/19 12:33:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_37db5d4fc7f80928a69639cc37500018 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:94e7d350-d132-4032-8bf2-e74bfdd84743", "USERNAME": "94e7d350-d132-4032-8bf2-e74bfdd84743", "PASSWORD": "a61dee56-879d-467b-8392-56ac7e32f07c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:33:20 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:33:20 asynchttp.go:292: Completed job 0b61b57ea0654a8b133a1c4c7f601b44 in 1.462366669s >[negroni] Started GET /queue/0b61b57ea0654a8b133a1c4c7f601b44 >[negroni] Completed 303 See Other in 159.841µs >[negroni] Started GET /blockvolumes/37db5d4fc7f80928a69639cc37500018 >[negroni] Completed 200 OK in 1.95885ms >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 847.945µs >[negroni] Started GET /volumes/00d7e51843ef621811a54f378d5dea3e >[negroni] Completed 200 OK in 852.119µs >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 498.752µs >[negroni] Started GET /volumes/8aad36048f6d364c329fa19445a48e92 >[negroni] Completed 200 OK in 547.126µs >[heketi] INFO 2018/08/19 12:34:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:34:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:34:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:34:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:34:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:34:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 38min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:34:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:34:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 17.186677ms >[asynchttp] INFO 2018/08/19 12:36:20 asynchttp.go:288: Started job 018f73a156d4c8d7f0fac5991f07389e >[heketi] INFO 2018/08/19 12:36:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/018f73a156d4c8d7f0fac5991f07389e >[negroni] Completed 200 OK in 127.132µs >[cmdexec] INFO 2018/08/19 12:36:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:36:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:36:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:36:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/018f73a156d4c8d7f0fac5991f07389e >[negroni] Completed 200 OK in 548.236µs >[negroni] Started GET /queue/018f73a156d4c8d7f0fac5991f07389e >[negroni] Completed 200 OK in 152.982µs >[negroni] Started GET /queue/018f73a156d4c8d7f0fac5991f07389e >[negroni] Completed 200 OK in 227.63µs >[kubeexec] DEBUG 2018/08/19 12:36:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_dc00030249e55f1fd3aecfd06617c194 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 85GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6cce2db7-c1cd-49d6-acbc-b064ec78322f", "USERNAME": "6cce2db7-c1cd-49d6-acbc-b064ec78322f", "PASSWORD": "9c3da4b9-ed37-4e85-a174-ee08a832b138", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:36:23 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:36:23 asynchttp.go:292: Completed job 018f73a156d4c8d7f0fac5991f07389e in 3.262047156s >[negroni] Started GET /queue/018f73a156d4c8d7f0fac5991f07389e >[negroni] Completed 303 See Other in 140.554µs >[negroni] Started GET /blockvolumes/dc00030249e55f1fd3aecfd06617c194 >[negroni] Completed 200 OK in 1.839823ms >[heketi] INFO 2018/08/19 12:36:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:36:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ4743 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:36:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:36:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ4816 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:36:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:36:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4812 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:36:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:36:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:36:50 Free size 2 is less than the requested block volume size 5 >[heketi] INFO 2018/08/19 12:36:50 Allocating brick set #0 >[negroni] Completed 202 Accepted in 53.422045ms >[asynchttp] INFO 2018/08/19 12:36:50 asynchttp.go:288: Started job ea215a30e2300bfcd9eac077e6841232 >[heketi] INFO 2018/08/19 12:36:50 Started async operation: Create Block Volume >[heketi] INFO 2018/08/19 12:36:50 Creating brick 1433d7e19161e1af9d1ceac56f32035a >[heketi] INFO 2018/08/19 12:36:50 Creating brick 10246ca80051bc6674ab548d648040d6 >[heketi] INFO 2018/08/19 12:36:50 Creating brick ef9430b4e5227d717f3321947cb741c9 >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 135.972µs >[kubeexec] DEBUG 2018/08/19 12:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_10246ca80051bc6674ab548d648040d6 >Result: >[kubeexec] DEBUG 2018/08/19 12:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_1433d7e19161e1af9d1ceac56f32035a >Result: >[kubeexec] DEBUG 2018/08/19 12:36:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9 >Result: >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:36:51 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:36:51 temporarily rejecting block volume request: pending block-hosting-volume found >[negroni] Completed 429 Too Many Requests in 2.768103ms >[heketi] ERROR 2018/08/19 12:36:51 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[kubeexec] DEBUG 2018/08/19 12:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_10246ca80051bc6674ab548d648040d6 --virtualsize 104857600K --name brick_10246ca80051bc6674ab548d648040d6 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_10246ca80051bc6674ab548d648040d6" created. >[kubeexec] DEBUG 2018/08/19 12:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_82e84d65676b46f68664c91c893d35d0 --virtualsize 104857600K --name brick_1433d7e19161e1af9d1ceac56f32035a >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_1433d7e19161e1af9d1ceac56f32035a" created. >[kubeexec] DEBUG 2018/08/19 12:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_ef9430b4e5227d717f3321947cb741c9 --virtualsize 104857600K --name brick_ef9430b4e5227d717f3321947cb741c9 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_ef9430b4e5227d717f3321947cb741c9" created. >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 142.248µs >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_10246ca80051bc6674ab548d648040d6 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_10246ca80051bc6674ab548d648040d6 isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 120.299µs >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9 isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_1433d7e19161e1af9d1ceac56f32035a >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_1433d7e19161e1af9d1ceac56f32035a isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_10246ca80051bc6674ab548d648040d6 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_10246ca80051bc6674ab548d648040d6 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_1433d7e19161e1af9d1ceac56f32035a /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_1433d7e19161e1af9d1ceac56f32035a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:36:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_10246ca80051bc6674ab548d648040d6 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_10246ca80051bc6674ab548d648040d6 >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_1433d7e19161e1af9d1ceac56f32035a /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_1433d7e19161e1af9d1ceac56f32035a >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9 >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_10246ca80051bc6674ab548d648040d6/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_1433d7e19161e1af9d1ceac56f32035a/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick >Result: >[cmdexec] INFO 2018/08/19 12:36:53 Creating volume vol_45a4ba9ee3f5a61061b35d95ae58adc8 replica 3 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:36:53 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:36:53 temporarily rejecting block volume request: pending block-hosting-volume found >[heketi] ERROR 2018/08/19 12:36:53 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[negroni] Completed 429 Too Many Requests in 2.504289ms >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 120.146µs >[kubeexec] DEBUG 2018/08/19 12:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_45a4ba9ee3f5a61061b35d95ae58adc8 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_1433d7e19161e1af9d1ceac56f32035a/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_10246ca80051bc6674ab548d648040d6/brick >Result: volume create: vol_45a4ba9ee3f5a61061b35d95ae58adc8: success: please start the volume to access data >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 131.569µs >[kubeexec] DEBUG 2018/08/19 12:36:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume set vol_45a4ba9ee3f5a61061b35d95ae58adc8 group gluster-block >Result: volume set: success >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 168.845µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:36:55 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:36:55 temporarily rejecting block volume request: pending block-hosting-volume found >[heketi] ERROR 2018/08/19 12:36:55 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[negroni] Completed 429 Too Many Requests in 1.657965ms >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 112.116µs >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 154.63µs >[kubeexec] DEBUG 2018/08/19 12:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_45a4ba9ee3f5a61061b35d95ae58adc8 >Result: volume start: vol_45a4ba9ee3f5a61061b35d95ae58adc8: success >[cmdexec] INFO 2018/08/19 12:36:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:36:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:36:58 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:36:58 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:36:58 temporarily rejecting block volume request: pending block-hosting-volume found >[negroni] Completed 429 Too Many Requests in 1.588932ms >[heketi] ERROR 2018/08/19 12:36:58 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[kubeexec] DEBUG 2018/08/19 12:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:36:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:36:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 200 OK in 178.437µs >[kubeexec] DEBUG 2018/08/19 12:36:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_977042bff747c432ba64322466a74cfd ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c24b1ddf-3fc7-4a39-b1be-cfe8e17511f3", "USERNAME": "c24b1ddf-3fc7-4a39-b1be-cfe8e17511f3", "PASSWORD": "6d524bec-7d51-41ce-8655-52a6fe2c4ca9", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:36:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:36:59 asynchttp.go:292: Completed job ea215a30e2300bfcd9eac077e6841232 in 9.000712985s >[negroni] Started GET /queue/ea215a30e2300bfcd9eac077e6841232 >[negroni] Completed 303 See Other in 132.142µs >[negroni] Started GET /blockvolumes/977042bff747c432ba64322466a74cfd >[negroni] Completed 200 OK in 1.365008ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:01 Free size 2 is less than the requested block volume size 5 >[asynchttp] INFO 2018/08/19 12:37:01 asynchttp.go:288: Started job 03df66de5eddc33a39d46bce9976b5ea >[heketi] INFO 2018/08/19 12:37:01 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 71.102157ms >[cmdexec] INFO 2018/08/19 12:37:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/03df66de5eddc33a39d46bce9976b5ea >[negroni] Completed 200 OK in 299.196µs >[kubeexec] DEBUG 2018/08/19 12:37:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:01 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/03df66de5eddc33a39d46bce9976b5ea >[negroni] Completed 200 OK in 156.427µs >[kubeexec] DEBUG 2018/08/19 12:37:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_0cc4cd96fd5d7b9ff7b76fd03a6eca1a ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c6cdf3fe-10a2-42ae-9f21-fb586d979ca1", "USERNAME": "c6cdf3fe-10a2-42ae-9f21-fb586d979ca1", "PASSWORD": "caea2475-17d9-439e-a502-6efb35b5d043", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:02 asynchttp.go:292: Completed job 03df66de5eddc33a39d46bce9976b5ea in 1.552820979s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:03 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 16.093216ms >[asynchttp] INFO 2018/08/19 12:37:03 asynchttp.go:288: Started job 21cda977c9ad6aeeea4e784e5577b53f >[heketi] INFO 2018/08/19 12:37:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/21cda977c9ad6aeeea4e784e5577b53f >[negroni] Completed 200 OK in 106.753µs >[cmdexec] INFO 2018/08/19 12:37:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:03 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/03df66de5eddc33a39d46bce9976b5ea >[negroni] Completed 303 See Other in 179.26µs >[negroni] Started GET /blockvolumes/0cc4cd96fd5d7b9ff7b76fd03a6eca1a >[negroni] Completed 200 OK in 729.823µs >[kubeexec] DEBUG 2018/08/19 12:37:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/21cda977c9ad6aeeea4e784e5577b53f >[negroni] Completed 200 OK in 125.721µs >[kubeexec] DEBUG 2018/08/19 12:37:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_0f7c9b41875bce6c6522dda1f94a9ee6 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4e522aee-269c-44e8-a553-9dfa9e17e999", "USERNAME": "4e522aee-269c-44e8-a553-9dfa9e17e999", "PASSWORD": "840cfdf4-7089-41bc-8db1-8d63927b7c93", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:04 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:04 asynchttp.go:292: Completed job 21cda977c9ad6aeeea4e784e5577b53f in 1.494777415s >[negroni] Started GET /queue/21cda977c9ad6aeeea4e784e5577b53f >[negroni] Completed 303 See Other in 134.891µs >[negroni] Started GET /blockvolumes/0f7c9b41875bce6c6522dda1f94a9ee6 >[negroni] Completed 200 OK in 1.979367ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:06 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 9.291972ms >[asynchttp] INFO 2018/08/19 12:37:06 asynchttp.go:288: Started job 81f5cd94a157bc9bd402feafc265ac42 >[heketi] INFO 2018/08/19 12:37:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/81f5cd94a157bc9bd402feafc265ac42 >[negroni] Completed 200 OK in 91.799µs >[cmdexec] INFO 2018/08/19 12:37:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:07 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/81f5cd94a157bc9bd402feafc265ac42 >[negroni] Completed 200 OK in 132.708µs >[kubeexec] DEBUG 2018/08/19 12:37:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_9381f35b40ad3d968df563142bf890aa ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4bb72d37-e787-495f-98c7-a5998c1c9d56", "USERNAME": "4bb72d37-e787-495f-98c7-a5998c1c9d56", "PASSWORD": "57ef1399-5e90-4009-a810-fdf0b5709dd6", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:08 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:08 asynchttp.go:292: Completed job 81f5cd94a157bc9bd402feafc265ac42 in 1.605872639s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:08 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 10.662961ms >[asynchttp] INFO 2018/08/19 12:37:08 asynchttp.go:288: Started job be9359efeec281a227e9b6ec68430c9d >[heketi] INFO 2018/08/19 12:37:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/be9359efeec281a227e9b6ec68430c9d >[negroni] Completed 200 OK in 111.634µs >[cmdexec] INFO 2018/08/19 12:37:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/81f5cd94a157bc9bd402feafc265ac42 >[negroni] Completed 303 See Other in 162.047µs >[kubeexec] DEBUG 2018/08/19 12:37:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /blockvolumes/9381f35b40ad3d968df563142bf890aa >[negroni] Completed 200 OK in 602.599µs >[kubeexec] DEBUG 2018/08/19 12:37:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/be9359efeec281a227e9b6ec68430c9d >[negroni] Completed 200 OK in 127.526µs >[kubeexec] DEBUG 2018/08/19 12:37:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_933aaf480a75edaf20c4c0183b213632 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:613980ee-a846-4306-8dd9-6c0168f86c43", "USERNAME": "613980ee-a846-4306-8dd9-6c0168f86c43", "PASSWORD": "df573cd1-544a-41f9-9d88-ce3f4ea7db97", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:10 asynchttp.go:292: Completed job be9359efeec281a227e9b6ec68430c9d in 1.872102926s >[negroni] Started GET /queue/be9359efeec281a227e9b6ec68430c9d >[negroni] Completed 303 See Other in 141.69µs >[negroni] Started GET /blockvolumes/933aaf480a75edaf20c4c0183b213632 >[negroni] Completed 200 OK in 2.292809ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:12 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 10.969614ms >[asynchttp] INFO 2018/08/19 12:37:12 asynchttp.go:288: Started job 79ffa01a9897c55da464e2331cf26e1a >[heketi] INFO 2018/08/19 12:37:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/79ffa01a9897c55da464e2331cf26e1a >[negroni] Completed 200 OK in 109.866µs >[cmdexec] INFO 2018/08/19 12:37:12 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:12 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/79ffa01a9897c55da464e2331cf26e1a >[negroni] Completed 200 OK in 183.635µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:14 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 9.936657ms >[asynchttp] INFO 2018/08/19 12:37:14 asynchttp.go:288: Started job 4d3cd6053c71365fdb0d150c740889c2 >[heketi] INFO 2018/08/19 12:37:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/4d3cd6053c71365fdb0d150c740889c2 >[negroni] Completed 200 OK in 112.794µs >[cmdexec] INFO 2018/08/19 12:37:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_7e79dab7aed560396bc8630a86bfe403 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:361ee558-efa8-4870-bf4d-eb0ea2bd8180", "USERNAME": "361ee558-efa8-4870-bf4d-eb0ea2bd8180", "PASSWORD": "f9d60a4c-c0d2-45ee-bf74-5df7b9416043", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 12:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:14 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/79ffa01a9897c55da464e2331cf26e1a >[negroni] Completed 200 OK in 162.117µs >[kubeexec] DEBUG 2018/08/19 12:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 12:37:14 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:14 asynchttp.go:292: Completed job 79ffa01a9897c55da464e2331cf26e1a in 2.11539558s >[kubeexec] DEBUG 2018/08/19 12:37:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4d3cd6053c71365fdb0d150c740889c2 >[negroni] Completed 200 OK in 152.925µs >[negroni] Started GET /queue/79ffa01a9897c55da464e2331cf26e1a >[negroni] Completed 303 See Other in 136.149µs >[negroni] Started GET /blockvolumes/7e79dab7aed560396bc8630a86bfe403 >[negroni] Completed 200 OK in 1.77544ms >[kubeexec] DEBUG 2018/08/19 12:37:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_7b7e896fa83a35604f8d5ae607b49cd8 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4cfaaa0b-f394-4870-bd9c-1003a07de2c6", "USERNAME": "4cfaaa0b-f394-4870-bd9c-1003a07de2c6", "PASSWORD": "eeaf4604-47f6-4d76-a48e-57f7d3772236", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:16 asynchttp.go:292: Completed job 4d3cd6053c71365fdb0d150c740889c2 in 1.676061745s >[negroni] Started GET /queue/4d3cd6053c71365fdb0d150c740889c2 >[negroni] Completed 303 See Other in 165.293µs >[negroni] Started GET /blockvolumes/7b7e896fa83a35604f8d5ae607b49cd8 >[negroni] Completed 200 OK in 1.512827ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:18 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 9.457534ms >[asynchttp] INFO 2018/08/19 12:37:18 asynchttp.go:288: Started job aaf6a0835cd5331c71059feffbeab05f >[heketi] INFO 2018/08/19 12:37:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/aaf6a0835cd5331c71059feffbeab05f >[negroni] Completed 200 OK in 92.992µs >[cmdexec] INFO 2018/08/19 12:37:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 41min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/aaf6a0835cd5331c71059feffbeab05f >[negroni] Completed 200 OK in 160.963µs >[kubeexec] DEBUG 2018/08/19 12:37:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_16fa08bd873c994ba86ebef232aa11ff ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4f6e5d00-beb3-4c84-ae98-5130b76cd494", "USERNAME": "4f6e5d00-beb3-4c84-ae98-5130b76cd494", "PASSWORD": "c41f83f1-d96a-4c0f-9ce8-68e9fb543894", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:19 asynchttp.go:292: Completed job aaf6a0835cd5331c71059feffbeab05f in 1.742723383s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:19 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 12.990196ms >[asynchttp] INFO 2018/08/19 12:37:19 asynchttp.go:288: Started job 6a9b1c5e7b7355109a2606371f629694 >[heketi] INFO 2018/08/19 12:37:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/6a9b1c5e7b7355109a2606371f629694 >[negroni] Completed 200 OK in 130.705µs >[cmdexec] INFO 2018/08/19 12:37:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/aaf6a0835cd5331c71059feffbeab05f >[negroni] Completed 303 See Other in 146.708µs >[negroni] Started GET /blockvolumes/16fa08bd873c994ba86ebef232aa11ff >[negroni] Completed 200 OK in 1.149692ms >[kubeexec] DEBUG 2018/08/19 12:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 41min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6a9b1c5e7b7355109a2606371f629694 >[negroni] Completed 200 OK in 139.486µs >[kubeexec] DEBUG 2018/08/19 12:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_eeb28dde662192a703d255fd2e53d0d7 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a52038a4-5d34-472e-9ba0-16258ca4bc96", "USERNAME": "a52038a4-5d34-472e-9ba0-16258ca4bc96", "PASSWORD": "96884d53-418d-46c7-95e0-500dd1b8ec5d", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:21 asynchttp.go:292: Completed job 6a9b1c5e7b7355109a2606371f629694 in 1.993836822s >[negroni] Started GET /queue/6a9b1c5e7b7355109a2606371f629694 >[negroni] Completed 303 See Other in 171.246µs >[negroni] Started GET /blockvolumes/eeb28dde662192a703d255fd2e53d0d7 >[negroni] Completed 200 OK in 1.344021ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:37:23 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 10.322054ms >[asynchttp] INFO 2018/08/19 12:37:23 asynchttp.go:288: Started job cc637e345ac044f52fb851fda134f7ae >[heketi] INFO 2018/08/19 12:37:23 Started async operation: Create Block Volume >[negroni] Started GET /queue/cc637e345ac044f52fb851fda134f7ae >[negroni] Completed 200 OK in 88.098µs >[cmdexec] INFO 2018/08/19 12:37:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:37:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:37:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 41min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:37:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cc637e345ac044f52fb851fda134f7ae >[negroni] Completed 200 OK in 149.911µs >[kubeexec] DEBUG 2018/08/19 12:37:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_82a1b738e81c902823a1c93b06dd35f9 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:eb068b1c-d283-41c4-8a6e-0d3b72f6d5e2", "USERNAME": "eb068b1c-d283-41c4-8a6e-0d3b72f6d5e2", "PASSWORD": "80571d9d-2dcd-4ebf-82de-459bb594bd54", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:37:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:37:25 asynchttp.go:292: Completed job cc637e345ac044f52fb851fda134f7ae in 1.781581814s >[negroni] Started GET /queue/cc637e345ac044f52fb851fda134f7ae >[negroni] Completed 303 See Other in 165.705µs >[negroni] Started GET /blockvolumes/82a1b738e81c902823a1c93b06dd35f9 >[negroni] Completed 200 OK in 1.781867ms >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 936.766µs >[negroni] Started GET /volumes/00d7e51843ef621811a54f378d5dea3e >[negroni] Completed 200 OK in 846.671µs >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 200 OK in 954.205µs >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 648.129µs >[negroni] Started GET /volumes/8aad36048f6d364c329fa19445a48e92 >[negroni] Completed 200 OK in 679.645µs >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 401 Unauthorized in 182.118µs >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 200 OK in 825.238µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:29 Free size 2 is less than the requested block volume size 40 >[negroni] Completed 202 Accepted in 27.575024ms >[asynchttp] INFO 2018/08/19 12:38:29 asynchttp.go:288: Started job 0453cd735967c3919125f5edfa8413fc >[heketi] INFO 2018/08/19 12:38:29 Started async operation: Create Block Volume >[negroni] Started GET /queue/0453cd735967c3919125f5edfa8413fc >[negroni] Completed 200 OK in 122.208µs >[cmdexec] INFO 2018/08/19 12:38:29 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:38:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:38:29 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:38:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0453cd735967c3919125f5edfa8413fc >[negroni] Completed 200 OK in 138.318µs >[negroni] Started GET /queue/0453cd735967c3919125f5edfa8413fc >[negroni] Completed 200 OK in 146.878µs >[kubeexec] DEBUG 2018/08/19 12:38:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_5efba45d201e343007e3849ad1997aa5 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 40GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f3345afc-bf0d-4bee-861d-92e1aa299c31", "USERNAME": "f3345afc-bf0d-4bee-861d-92e1aa299c31", "PASSWORD": "fc43008a-a08c-4532-91bb-3dc30feb8e5d", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:38:32 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:38:32 asynchttp.go:292: Completed job 0453cd735967c3919125f5edfa8413fc in 2.489800707s >[negroni] Started GET /queue/0453cd735967c3919125f5edfa8413fc >[negroni] Completed 303 See Other in 167.499µs >[negroni] Started GET /blockvolumes/5efba45d201e343007e3849ad1997aa5 >[negroni] Completed 200 OK in 1.822812ms >[heketi] INFO 2018/08/19 12:38:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:38:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:38:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:38:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:38:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:38:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:38:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:38:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:42 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 19.631593ms >[asynchttp] INFO 2018/08/19 12:38:42 asynchttp.go:288: Started job f38a9f9c1064b897f0ff32af1af752e6 >[heketi] INFO 2018/08/19 12:38:42 Started async operation: Create Block Volume >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f38a9f9c1064b897f0ff32af1af752e6 >[negroni] Completed 200 OK in 164.607µs >[kubeexec] DEBUG 2018/08/19 12:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:42 Free size 2 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 12.351975ms >[asynchttp] INFO 2018/08/19 12:38:42 asynchttp.go:288: Started job 1e568035fb380f4b97dc3298370e4bdd >[heketi] INFO 2018/08/19 12:38:42 Started async operation: Create Block Volume >[negroni] Started GET /queue/1e568035fb380f4b97dc3298370e4bdd >[negroni] Completed 200 OK in 90.582µs >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6230 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:38:42 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6182 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:38:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f38a9f9c1064b897f0ff32af1af752e6 >[negroni] Completed 200 OK in 170.716µs >[negroni] Started GET /queue/1e568035fb380f4b97dc3298370e4bdd >[negroni] Completed 200 OK in 168.689µs >[kubeexec] DEBUG 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_3ba48c0b9bc39e3c4cfc1ee4e667c9a6 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f2543ec0-d759-4bdb-9879-7a280833153f", "USERNAME": "f2543ec0-d759-4bdb-9879-7a280833153f", "PASSWORD": "3495c54b-1914-4e21-a4f2-239aad4e7ffa", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:38:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:38:44 asynchttp.go:292: Completed job f38a9f9c1064b897f0ff32af1af752e6 in 1.987716341s >[negroni] Started GET /queue/f38a9f9c1064b897f0ff32af1af752e6 >[negroni] Completed 303 See Other in 196.634µs >[negroni] Started GET /blockvolumes/3ba48c0b9bc39e3c4cfc1ee4e667c9a6 >[negroni] Completed 200 OK in 1.872378ms >[kubeexec] DEBUG 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:38:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 5GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 28]: Stdout [{ "RESULT": "FAIL", "errCode": 28, "errMsg": "Not able to create storage for vol_45a4ba9ee3f5a61061b35d95ae58adc8\/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 [No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 2]: Stdout [{ "RESULT": "FAIL", "errCode": 2, "errMsg": "block vol_45a4ba9ee3f5a61061b35d95ae58adc8\/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 doesn't exist" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 12:38:44 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 12:38:44 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 12:38:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1e568035fb380f4b97dc3298370e4bdd >[negroni] Completed 200 OK in 183.77µs >[kubeexec] DEBUG 2018/08/19 12:38:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ6351 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 12:38:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 2]: Stdout [{ "RESULT": "FAIL", "errCode": 2, "errMsg": "block vol_45a4ba9ee3f5a61061b35d95ae58adc8\/blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3 doesn't exist" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 12:38:45 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_97e26c17e2ce5fa0467c0bd6bd1bbfa3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 12:38:45 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 12:38:45 asynchttp.go:292: Completed job 1e568035fb380f4b97dc3298370e4bdd in 2.605726492s >[heketi] ERROR 2018/08/19 12:38:45 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/1e568035fb380f4b97dc3298370e4bdd >[negroni] Completed 500 Internal Server Error in 213.098µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:46 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:46 Free size 3 is less than the requested block volume size 5 >[heketi] INFO 2018/08/19 12:38:46 Allocating brick set #0 >[negroni] Completed 202 Accepted in 82.946033ms >[asynchttp] INFO 2018/08/19 12:38:46 asynchttp.go:288: Started job 9a0899e33ba6f9ed1c5dacd192c69558 >[heketi] INFO 2018/08/19 12:38:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 179.904µs >[heketi] INFO 2018/08/19 12:38:46 Creating brick 61f23fbd999cc0c1ebaf4b7f05121748 >[heketi] INFO 2018/08/19 12:38:46 Creating brick 73ccc3351fe2b8705b443f0bbe3ed284 >[heketi] INFO 2018/08/19 12:38:46 Creating brick 86bfa8254e302837df56865b8dc444ed >[kubeexec] DEBUG 2018/08/19 12:38:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_73ccc3351fe2b8705b443f0bbe3ed284 >Result: >[kubeexec] DEBUG 2018/08/19 12:38:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_61f23fbd999cc0c1ebaf4b7f05121748 >Result: >[kubeexec] DEBUG 2018/08/19 12:38:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_86bfa8254e302837df56865b8dc444ed >Result: >[kubeexec] DEBUG 2018/08/19 12:38:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_73ccc3351fe2b8705b443f0bbe3ed284 --virtualsize 104857600K --name brick_73ccc3351fe2b8705b443f0bbe3ed284 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_73ccc3351fe2b8705b443f0bbe3ed284" created. >[kubeexec] DEBUG 2018/08/19 12:38:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_f8759c7a4bf238ce8e2b132984686ee2 --virtualsize 104857600K --name brick_61f23fbd999cc0c1ebaf4b7f05121748 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_61f23fbd999cc0c1ebaf4b7f05121748" created. >[kubeexec] DEBUG 2018/08/19 12:38:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 524288K --chunksize 256K --size 104857600K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_86bfa8254e302837df56865b8dc444ed --virtualsize 104857600K --name brick_86bfa8254e302837df56865b8dc444ed >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_86bfa8254e302837df56865b8dc444ed" created. >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 187.164µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:47 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:47 Free size 3 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:47 temporarily rejecting block volume request: pending block-hosting-volume found >[heketi] ERROR 2018/08/19 12:38:47 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[negroni] Completed 429 Too Many Requests in 3.210586ms >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 132.564µs >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_86bfa8254e302837df56865b8dc444ed >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_86bfa8254e302837df56865b8dc444ed isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_86bfa8254e302837df56865b8dc444ed /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_86bfa8254e302837df56865b8dc444ed xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_61f23fbd999cc0c1ebaf4b7f05121748 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_61f23fbd999cc0c1ebaf4b7f05121748 isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_73ccc3351fe2b8705b443f0bbe3ed284 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_73ccc3351fe2b8705b443f0bbe3ed284 isize=512 agcount=16, agsize=1638400 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=26214400, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=12800, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_61f23fbd999cc0c1ebaf4b7f05121748 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_61f23fbd999cc0c1ebaf4b7f05121748 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_73ccc3351fe2b8705b443f0bbe3ed284 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_73ccc3351fe2b8705b443f0bbe3ed284 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:38:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_86bfa8254e302837df56865b8dc444ed /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_86bfa8254e302837df56865b8dc444ed >Result: >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_86bfa8254e302837df56865b8dc444ed/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_73ccc3351fe2b8705b443f0bbe3ed284 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_73ccc3351fe2b8705b443f0bbe3ed284 >Result: >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_73ccc3351fe2b8705b443f0bbe3ed284/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_61f23fbd999cc0c1ebaf4b7f05121748 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_61f23fbd999cc0c1ebaf4b7f05121748 >Result: >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_61f23fbd999cc0c1ebaf4b7f05121748/brick >Result: >[cmdexec] INFO 2018/08/19 12:38:49 Creating volume vol_3589da219d6536edb00cf7b533976e25 replica 3 >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 164.546µs >[kubeexec] DEBUG 2018/08/19 12:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: gluster --mode=script volume create vol_3589da219d6536edb00cf7b533976e25 replica 3 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_61f23fbd999cc0c1ebaf4b7f05121748/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_73ccc3351fe2b8705b443f0bbe3ed284/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_86bfa8254e302837df56865b8dc444ed/brick >Result: volume create: vol_3589da219d6536edb00cf7b533976e25: success: please start the volume to access data >[negroni] Started POST /blockvolumes >[heketi] ERROR 2018/08/19 12:38:50 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[heketi] WARNING 2018/08/19 12:38:50 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:50 Free size 3 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:50 temporarily rejecting block volume request: pending block-hosting-volume found >[negroni] Completed 429 Too Many Requests in 1.924133ms >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 112.837µs >[kubeexec] DEBUG 2018/08/19 12:38:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: gluster --mode=script volume set vol_3589da219d6536edb00cf7b533976e25 group gluster-block >Result: volume set: success >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 155.637µs >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 195.224µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:52 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:52 Free size 3 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:52 temporarily rejecting block volume request: pending block-hosting-volume found >[heketi] ERROR 2018/08/19 12:38:52 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[negroni] Completed 429 Too Many Requests in 3.543847ms >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 147.088µs >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 142.721µs >[kubeexec] DEBUG 2018/08/19 12:38:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: gluster --mode=script volume start vol_3589da219d6536edb00cf7b533976e25 >Result: volume start: vol_3589da219d6536edb00cf7b533976e25: success >[cmdexec] INFO 2018/08/19 12:38:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:38:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:55 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:55 Free size 3 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:55 temporarily rejecting block volume request: pending block-hosting-volume found >[heketi] ERROR 2018/08/19 12:38:55 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Block Volume Build Failed: Server handling too many operations >[negroni] Completed 429 Too Many Requests in 2.143588ms >[kubeexec] DEBUG 2018/08/19 12:38:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:38:55 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:38:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 159.114µs >[kubeexec] DEBUG 2018/08/19 12:38:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 200 OK in 138.159µs >[kubeexec] DEBUG 2018/08/19 12:38:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_97cf559b4833e868b22204919b861aff ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:026fcc8d-a723-416f-9bca-0ca9837aedfb", "USERNAME": "026fcc8d-a723-416f-9bca-0ca9837aedfb", "PASSWORD": "e6a81aeb-8d6a-4c89-8ef6-3ef5495b417c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:38:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:38:57 asynchttp.go:292: Completed job 9a0899e33ba6f9ed1c5dacd192c69558 in 10.781798381s >[negroni] Started GET /queue/9a0899e33ba6f9ed1c5dacd192c69558 >[negroni] Completed 303 See Other in 250.988µs >[negroni] Started GET /blockvolumes/97cf559b4833e868b22204919b861aff >[negroni] Completed 200 OK in 1.789416ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:38:57 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:38:57 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 10.598798ms >[asynchttp] INFO 2018/08/19 12:38:57 asynchttp.go:288: Started job e6d5bdd08146e9c8053c35a0bb039b0b >[heketi] INFO 2018/08/19 12:38:57 Started async operation: Create Block Volume >[negroni] Started GET /queue/e6d5bdd08146e9c8053c35a0bb039b0b >[negroni] Completed 200 OK in 86.375µs >[cmdexec] INFO 2018/08/19 12:38:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:38:57 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:38:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:38:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:38:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e6d5bdd08146e9c8053c35a0bb039b0b >[negroni] Completed 200 OK in 170.033µs >[kubeexec] DEBUG 2018/08/19 12:38:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d02ba85022744725d25492913f121e0c ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3896a41d-19fa-4b1a-adac-4ae4de1074ed", "USERNAME": "3896a41d-19fa-4b1a-adac-4ae4de1074ed", "PASSWORD": "5cabeee1-797e-4371-9e27-70fa6807b151", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:38:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:38:59 asynchttp.go:292: Completed job e6d5bdd08146e9c8053c35a0bb039b0b in 1.977735862s >[negroni] Started GET /queue/e6d5bdd08146e9c8053c35a0bb039b0b >[negroni] Completed 303 See Other in 126.046µs >[negroni] Started GET /blockvolumes/d02ba85022744725d25492913f121e0c >[negroni] Completed 200 OK in 2.1023ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:00 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:00 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 9.430645ms >[asynchttp] INFO 2018/08/19 12:39:00 asynchttp.go:288: Started job c2f14ab954c84eb706781ecbb4d464b9 >[heketi] INFO 2018/08/19 12:39:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/c2f14ab954c84eb706781ecbb4d464b9 >[negroni] Completed 200 OK in 106.11µs >[cmdexec] INFO 2018/08/19 12:39:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:00 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:00 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c2f14ab954c84eb706781ecbb4d464b9 >[negroni] Completed 200 OK in 130.36µs >[negroni] Started GET /queue/c2f14ab954c84eb706781ecbb4d464b9 >[negroni] Completed 200 OK in 165.894µs >[kubeexec] DEBUG 2018/08/19 12:39:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f7c458a082c030ea6dee3f03f7f00e63 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c046fdbc-24d1-4380-8609-c47566da5572", "USERNAME": "c046fdbc-24d1-4380-8609-c47566da5572", "PASSWORD": "0a416e80-3a86-4fb0-bec4-91991c04358a", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:02 asynchttp.go:292: Completed job c2f14ab954c84eb706781ecbb4d464b9 in 2.155916109s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:03 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:03 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 11.614347ms >[asynchttp] INFO 2018/08/19 12:39:03 asynchttp.go:288: Started job 51847bf4b5e962e845978c134a421648 >[heketi] INFO 2018/08/19 12:39:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/51847bf4b5e962e845978c134a421648 >[negroni] Completed 200 OK in 92.653µs >[cmdexec] INFO 2018/08/19 12:39:03 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c2f14ab954c84eb706781ecbb4d464b9 >[negroni] Completed 303 See Other in 168.555µs >[negroni] Started GET /blockvolumes/f7c458a082c030ea6dee3f03f7f00e63 >[negroni] Completed 200 OK in 520.879µs >[negroni] Started GET /queue/51847bf4b5e962e845978c134a421648 >[negroni] Completed 200 OK in 173.789µs >[kubeexec] DEBUG 2018/08/19 12:39:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0a2a601086be6888c9fd939a8eb4be54 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a9159be6-e8cf-4bc7-9333-4f18c8f6a754", "USERNAME": "a9159be6-e8cf-4bc7-9333-4f18c8f6a754", "PASSWORD": "771ac83b-cf2f-4a19-a416-51e0161d549a", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:05 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:05 asynchttp.go:292: Completed job 51847bf4b5e962e845978c134a421648 in 1.998471095s >[negroni] Started GET /queue/51847bf4b5e962e845978c134a421648 >[negroni] Completed 303 See Other in 120.187µs >[negroni] Started GET /blockvolumes/0a2a601086be6888c9fd939a8eb4be54 >[negroni] Completed 200 OK in 1.57484ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:06 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:06 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 9.605038ms >[asynchttp] INFO 2018/08/19 12:39:06 asynchttp.go:288: Started job 403d0408b39435fe6aaca316f5728bca >[heketi] INFO 2018/08/19 12:39:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/403d0408b39435fe6aaca316f5728bca >[negroni] Completed 200 OK in 79.952µs >[cmdexec] INFO 2018/08/19 12:39:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:06 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/403d0408b39435fe6aaca316f5728bca >[negroni] Completed 200 OK in 143.917µs >[negroni] Started GET /queue/403d0408b39435fe6aaca316f5728bca >[negroni] Completed 200 OK in 171.149µs >[kubeexec] DEBUG 2018/08/19 12:39:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_afe51517f2a88bcde2d354786fcd6dcd ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e6bea676-1e61-455a-966f-4ddda8fd8cc1", "USERNAME": "e6bea676-1e61-455a-966f-4ddda8fd8cc1", "PASSWORD": "0b052bd8-441a-4211-af51-52d7c21e6321", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:08 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:08 asynchttp.go:292: Completed job 403d0408b39435fe6aaca316f5728bca in 2.080724619s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:08 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:08 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 11.057977ms >[asynchttp] INFO 2018/08/19 12:39:08 asynchttp.go:288: Started job 1569ededaf6ffc95697b0cd9983161cb >[heketi] INFO 2018/08/19 12:39:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/1569ededaf6ffc95697b0cd9983161cb >[negroni] Completed 200 OK in 97.339µs >[cmdexec] INFO 2018/08/19 12:39:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/403d0408b39435fe6aaca316f5728bca >[negroni] Completed 303 See Other in 239.007µs >[negroni] Started GET /blockvolumes/afe51517f2a88bcde2d354786fcd6dcd >[negroni] Completed 200 OK in 1.072815ms >[negroni] Started GET /queue/1569ededaf6ffc95697b0cd9983161cb >[negroni] Completed 200 OK in 134.06µs >[negroni] Started GET /queue/1569ededaf6ffc95697b0cd9983161cb >[negroni] Completed 200 OK in 144.791µs >[kubeexec] DEBUG 2018/08/19 12:39:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d6d094d080681a88eb8a463df75913a5 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:0b79a1a1-03fd-4beb-9be3-bb7256591126", "USERNAME": "0b79a1a1-03fd-4beb-9be3-bb7256591126", "PASSWORD": "c97d0739-81ae-48b7-99fc-5adfb8e169c6", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:10 asynchttp.go:292: Completed job 1569ededaf6ffc95697b0cd9983161cb in 2.019564331s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:11 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:11 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 12.786832ms >[asynchttp] INFO 2018/08/19 12:39:11 asynchttp.go:288: Started job 888f539eb88a5a4209720db8b0610bd8 >[heketi] INFO 2018/08/19 12:39:11 Started async operation: Create Block Volume >[negroni] Started GET /queue/888f539eb88a5a4209720db8b0610bd8 >[negroni] Completed 200 OK in 135.108µs >[cmdexec] INFO 2018/08/19 12:39:11 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1569ededaf6ffc95697b0cd9983161cb >[negroni] Completed 303 See Other in 169.763µs >[negroni] Started GET /blockvolumes/d6d094d080681a88eb8a463df75913a5 >[negroni] Completed 200 OK in 517.259µs >[kubeexec] DEBUG 2018/08/19 12:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/888f539eb88a5a4209720db8b0610bd8 >[negroni] Completed 200 OK in 134.255µs >[negroni] Started GET /queue/888f539eb88a5a4209720db8b0610bd8 >[negroni] Completed 200 OK in 113.017µs >[kubeexec] DEBUG 2018/08/19 12:39:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fe94ebd58bfd864d039f96b9f6849940 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:439fbd88-e6f9-46e1-b82a-886685fba0c0", "USERNAME": "439fbd88-e6f9-46e1-b82a-886685fba0c0", "PASSWORD": "acaaa33b-11cc-4013-8724-9bb86c43e6f0", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:13 asynchttp.go:292: Completed job 888f539eb88a5a4209720db8b0610bd8 in 2.297280071s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:14 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:14 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 11.470389ms >[asynchttp] INFO 2018/08/19 12:39:14 asynchttp.go:288: Started job 0bc4b531eed0269ba7157f2713c6d69c >[heketi] INFO 2018/08/19 12:39:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/0bc4b531eed0269ba7157f2713c6d69c >[negroni] Completed 200 OK in 122.462µs >[cmdexec] INFO 2018/08/19 12:39:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:14 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/888f539eb88a5a4209720db8b0610bd8 >[negroni] Completed 303 See Other in 165.714µs >[negroni] Started GET /blockvolumes/fe94ebd58bfd864d039f96b9f6849940 >[negroni] Completed 200 OK in 553.353µs >[kubeexec] DEBUG 2018/08/19 12:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0bc4b531eed0269ba7157f2713c6d69c >[negroni] Completed 200 OK in 155.642µs >[negroni] Started GET /queue/0bc4b531eed0269ba7157f2713c6d69c >[negroni] Completed 200 OK in 208.19µs >[kubeexec] DEBUG 2018/08/19 12:39:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ba51b81fbb61e6240b61fd205cb357c3 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:52d763c0-36fe-4067-857e-c2b9b958030e", "USERNAME": "52d763c0-36fe-4067-857e-c2b9b958030e", "PASSWORD": "e5deb87c-d8cb-4b0e-88f5-ab8fde1579ab", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:16 asynchttp.go:292: Completed job 0bc4b531eed0269ba7157f2713c6d69c in 2.114083681s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:17 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:17 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 73.546184ms >[asynchttp] INFO 2018/08/19 12:39:17 asynchttp.go:288: Started job 53d7e975293c70266669bb7b1ffd8a3e >[heketi] INFO 2018/08/19 12:39:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/53d7e975293c70266669bb7b1ffd8a3e >[negroni] Completed 200 OK in 128.4µs >[cmdexec] INFO 2018/08/19 12:39:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0bc4b531eed0269ba7157f2713c6d69c >[negroni] Completed 303 See Other in 206.609µs >[negroni] Started GET /blockvolumes/ba51b81fbb61e6240b61fd205cb357c3 >[negroni] Completed 200 OK in 714.213µs >[kubeexec] DEBUG 2018/08/19 12:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 42min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/53d7e975293c70266669bb7b1ffd8a3e >[negroni] Completed 200 OK in 156.246µs >[negroni] Started GET /queue/53d7e975293c70266669bb7b1ffd8a3e >[negroni] Completed 200 OK in 132.277µs >[kubeexec] DEBUG 2018/08/19 12:39:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_41e2f7bdf689a4483fdc616b625c5f70 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f9a37322-8529-495c-bec9-68de0fc758ac", "USERNAME": "f9a37322-8529-495c-bec9-68de0fc758ac", "PASSWORD": "9c07be6a-3725-47a0-baf7-b64040a80f52", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:19 asynchttp.go:292: Completed job 53d7e975293c70266669bb7b1ffd8a3e in 2.130798198s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:19 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:19 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 10.853927ms >[asynchttp] INFO 2018/08/19 12:39:19 asynchttp.go:288: Started job 5bc180a058520b3ad391fae1c7db4366 >[heketi] INFO 2018/08/19 12:39:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/5bc180a058520b3ad391fae1c7db4366 >[negroni] Completed 200 OK in 73.085µs >[cmdexec] INFO 2018/08/19 12:39:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/53d7e975293c70266669bb7b1ffd8a3e >[negroni] Completed 303 See Other in 204.367µs >[negroni] Started GET /blockvolumes/41e2f7bdf689a4483fdc616b625c5f70 >[negroni] Completed 200 OK in 944.106µs >[kubeexec] DEBUG 2018/08/19 12:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5bc180a058520b3ad391fae1c7db4366 >[negroni] Completed 200 OK in 119.137µs >[negroni] Started GET /queue/5bc180a058520b3ad391fae1c7db4366 >[negroni] Completed 200 OK in 130.436µs >[kubeexec] DEBUG 2018/08/19 12:39:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_50d9ab52accb141c90d87a2e76fa2b10 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6510f6ec-6865-4c60-a30d-94e6546464b7", "USERNAME": "6510f6ec-6865-4c60-a30d-94e6546464b7", "PASSWORD": "b122e11b-5346-40f0-8d7f-f3fdcd716415", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:22 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:22 asynchttp.go:292: Completed job 5bc180a058520b3ad391fae1c7db4366 in 2.384774365s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:22 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:22 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 11.984513ms >[asynchttp] INFO 2018/08/19 12:39:22 asynchttp.go:288: Started job 91e8a36afb2445fc0eee982a81d29b8d >[heketi] INFO 2018/08/19 12:39:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/91e8a36afb2445fc0eee982a81d29b8d >[negroni] Completed 200 OK in 92.962µs >[cmdexec] INFO 2018/08/19 12:39:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:22 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/5bc180a058520b3ad391fae1c7db4366 >[negroni] Completed 303 See Other in 138.544µs >[negroni] Started GET /blockvolumes/50d9ab52accb141c90d87a2e76fa2b10 >[negroni] Completed 200 OK in 650.483µs >[kubeexec] DEBUG 2018/08/19 12:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/91e8a36afb2445fc0eee982a81d29b8d >[negroni] Completed 200 OK in 170.723µs >[negroni] Started GET /queue/91e8a36afb2445fc0eee982a81d29b8d >[negroni] Completed 200 OK in 173.77µs >[kubeexec] DEBUG 2018/08/19 12:39:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d78ae8c6fdbf17f6f97bdafa3aa8f124 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:43e63d8c-73dd-4e87-964b-0ca696c79ba3", "USERNAME": "43e63d8c-73dd-4e87-964b-0ca696c79ba3", "PASSWORD": "b533c581-1da9-4986-975b-13864a0fad3c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:25 asynchttp.go:292: Completed job 91e8a36afb2445fc0eee982a81d29b8d in 2.310656219s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:25 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:25 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 14.147288ms >[asynchttp] INFO 2018/08/19 12:39:25 asynchttp.go:288: Started job 320fc1ef138dda6ce05e92e3e581dd1e >[heketi] INFO 2018/08/19 12:39:25 Started async operation: Create Block Volume >[negroni] Started GET /queue/320fc1ef138dda6ce05e92e3e581dd1e >[negroni] Completed 200 OK in 117.237µs >[cmdexec] INFO 2018/08/19 12:39:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:25 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/91e8a36afb2445fc0eee982a81d29b8d >[negroni] Completed 303 See Other in 154.723µs >[negroni] Started GET /blockvolumes/d78ae8c6fdbf17f6f97bdafa3aa8f124 >[negroni] Completed 200 OK in 1.299022ms >[kubeexec] DEBUG 2018/08/19 12:39:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:25 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/320fc1ef138dda6ce05e92e3e581dd1e >[negroni] Completed 200 OK in 127.361µs >[negroni] Started GET /queue/320fc1ef138dda6ce05e92e3e581dd1e >[negroni] Completed 200 OK in 133.457µs >[kubeexec] DEBUG 2018/08/19 12:39:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fb12b37f20dd08a9e0813d1195bf04bb ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4c2c52f1-48b4-4430-b54c-5ac697eb9fe8", "USERNAME": "4c2c52f1-48b4-4430-b54c-5ac697eb9fe8", "PASSWORD": "e41d54e2-edfe-4681-9d60-95f9e13fd344", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:27 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:27 asynchttp.go:292: Completed job 320fc1ef138dda6ce05e92e3e581dd1e in 2.320462667s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:28 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:28 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 18.08417ms >[asynchttp] INFO 2018/08/19 12:39:28 asynchttp.go:288: Started job 9d75ac87a34efa6accbb37d239d014c6 >[heketi] INFO 2018/08/19 12:39:28 Started async operation: Create Block Volume >[negroni] Started GET /queue/9d75ac87a34efa6accbb37d239d014c6 >[negroni] Completed 200 OK in 112.736µs >[cmdexec] INFO 2018/08/19 12:39:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/320fc1ef138dda6ce05e92e3e581dd1e >[negroni] Completed 303 See Other in 145.771µs >[negroni] Started GET /blockvolumes/fb12b37f20dd08a9e0813d1195bf04bb >[negroni] Completed 200 OK in 1.55271ms >[kubeexec] DEBUG 2018/08/19 12:39:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:28 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9d75ac87a34efa6accbb37d239d014c6 >[negroni] Completed 200 OK in 136.034µs >[negroni] Started GET /queue/9d75ac87a34efa6accbb37d239d014c6 >[negroni] Completed 200 OK in 203.888µs >[kubeexec] DEBUG 2018/08/19 12:39:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_b8ba116091f1049a3986ebd85f59d0dd ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d56edca0-7323-4bf0-9802-9b34192c84a7", "USERNAME": "d56edca0-7323-4bf0-9802-9b34192c84a7", "PASSWORD": "5dd521a5-f0e6-476c-87c8-ac98d289f6e9", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:30 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:30 asynchttp.go:292: Completed job 9d75ac87a34efa6accbb37d239d014c6 in 2.333877458s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:31 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:31 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 22.40395ms >[asynchttp] INFO 2018/08/19 12:39:31 asynchttp.go:288: Started job a18bb75b3c5fab6f0b1b8a34158666b8 >[heketi] INFO 2018/08/19 12:39:31 Started async operation: Create Block Volume >[negroni] Started GET /queue/a18bb75b3c5fab6f0b1b8a34158666b8 >[negroni] Completed 200 OK in 130.643µs >[cmdexec] INFO 2018/08/19 12:39:31 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9d75ac87a34efa6accbb37d239d014c6 >[negroni] Completed 303 See Other in 134.807µs >[negroni] Started GET /blockvolumes/b8ba116091f1049a3986ebd85f59d0dd >[negroni] Completed 200 OK in 1.677363ms >[kubeexec] DEBUG 2018/08/19 12:39:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:31 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a18bb75b3c5fab6f0b1b8a34158666b8 >[negroni] Completed 200 OK in 190.13µs >[negroni] Started GET /queue/a18bb75b3c5fab6f0b1b8a34158666b8 >[negroni] Completed 200 OK in 117.481µs >[kubeexec] DEBUG 2018/08/19 12:39:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_31c27994f63ef3d98cc077c3e21428e9 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:11fd8157-680c-41ff-8d4e-706886b6dc05", "USERNAME": "11fd8157-680c-41ff-8d4e-706886b6dc05", "PASSWORD": "863dcc0a-9d93-4d86-9141-e30f0938fe55", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:33 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:33 asynchttp.go:292: Completed job a18bb75b3c5fab6f0b1b8a34158666b8 in 2.576393927s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:33 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:33 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 16.502584ms >[asynchttp] INFO 2018/08/19 12:39:33 asynchttp.go:288: Started job 75e99818a11f3e09fe1c5e2281493ef2 >[heketi] INFO 2018/08/19 12:39:33 Started async operation: Create Block Volume >[negroni] Started GET /queue/75e99818a11f3e09fe1c5e2281493ef2 >[negroni] Completed 200 OK in 104.818µs >[cmdexec] INFO 2018/08/19 12:39:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:34 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a18bb75b3c5fab6f0b1b8a34158666b8 >[negroni] Completed 303 See Other in 159.685µs >[negroni] Started GET /blockvolumes/31c27994f63ef3d98cc077c3e21428e9 >[negroni] Completed 200 OK in 1.54728ms >[kubeexec] DEBUG 2018/08/19 12:39:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:34 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/75e99818a11f3e09fe1c5e2281493ef2 >[negroni] Completed 200 OK in 131.001µs >[negroni] Started GET /queue/75e99818a11f3e09fe1c5e2281493ef2 >[negroni] Completed 200 OK in 156.168µs >[kubeexec] DEBUG 2018/08/19 12:39:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_badbf77121230eefaf245e08553ab5a5 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d7ba5268-a0cc-480b-8398-38b0e236638a", "USERNAME": "d7ba5268-a0cc-480b-8398-38b0e236638a", "PASSWORD": "48ba3220-d97e-4ffe-a9e1-950234c9db30", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:36 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:36 asynchttp.go:292: Completed job 75e99818a11f3e09fe1c5e2281493ef2 in 2.568573906s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:36 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:36 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 16.200611ms >[asynchttp] INFO 2018/08/19 12:39:36 asynchttp.go:288: Started job 780c2a3a3a1e9885b810568d0f68e30b >[heketi] INFO 2018/08/19 12:39:36 Started async operation: Create Block Volume >[negroni] Started GET /queue/780c2a3a3a1e9885b810568d0f68e30b >[negroni] Completed 200 OK in 99.384µs >[cmdexec] INFO 2018/08/19 12:39:36 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/75e99818a11f3e09fe1c5e2281493ef2 >[negroni] Completed 303 See Other in 164.991µs >[negroni] Started GET /blockvolumes/badbf77121230eefaf245e08553ab5a5 >[negroni] Completed 200 OK in 2.007687ms >[kubeexec] DEBUG 2018/08/19 12:39:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/780c2a3a3a1e9885b810568d0f68e30b >[negroni] Completed 200 OK in 126.842µs >[negroni] Started GET /queue/780c2a3a3a1e9885b810568d0f68e30b >[negroni] Completed 200 OK in 118.873µs >[kubeexec] DEBUG 2018/08/19 12:39:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c75e354b772e6037920e02b5abc103ea ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:34f1fd15-05e4-4e0b-a5ea-8a2a95fa32ad", "USERNAME": "34f1fd15-05e4-4e0b-a5ea-8a2a95fa32ad", "PASSWORD": "63b2e414-de94-47ea-9782-5e4bf74f96ee", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:39 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:39 asynchttp.go:292: Completed job 780c2a3a3a1e9885b810568d0f68e30b in 2.49726915s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:39 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:39 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 23.355249ms >[asynchttp] INFO 2018/08/19 12:39:39 asynchttp.go:288: Started job 4afa40341249340f7794b81fdbacad37 >[heketi] INFO 2018/08/19 12:39:39 Started async operation: Create Block Volume >[negroni] Started GET /queue/4afa40341249340f7794b81fdbacad37 >[negroni] Completed 200 OK in 118.923µs >[cmdexec] INFO 2018/08/19 12:39:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:39 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/780c2a3a3a1e9885b810568d0f68e30b >[negroni] Completed 303 See Other in 139.92µs >[negroni] Started GET /blockvolumes/c75e354b772e6037920e02b5abc103ea >[negroni] Completed 200 OK in 1.341739ms >[kubeexec] DEBUG 2018/08/19 12:39:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:39 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4afa40341249340f7794b81fdbacad37 >[negroni] Completed 200 OK in 136.616µs >[negroni] Started GET /queue/4afa40341249340f7794b81fdbacad37 >[negroni] Completed 200 OK in 111.569µs >[kubeexec] DEBUG 2018/08/19 12:39:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_dc2ee508e0a489300de3025b6126876f ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bfad83e4-9e3f-4504-a72f-cffa694ac0b3", "USERNAME": "bfad83e4-9e3f-4504-a72f-cffa694ac0b3", "PASSWORD": "36688d05-d7d3-4063-90ac-c08ceec49513", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:42 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:42 asynchttp.go:292: Completed job 4afa40341249340f7794b81fdbacad37 in 2.539868718s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:42 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:42 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 52.926197ms >[asynchttp] INFO 2018/08/19 12:39:42 asynchttp.go:288: Started job 3f06586c6e46a53541a8fe1be8f101fa >[heketi] INFO 2018/08/19 12:39:42 Started async operation: Create Block Volume >[negroni] Started GET /queue/3f06586c6e46a53541a8fe1be8f101fa >[negroni] Completed 200 OK in 127.793µs >[cmdexec] INFO 2018/08/19 12:39:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4afa40341249340f7794b81fdbacad37 >[negroni] Completed 303 See Other in 132.806µs >[negroni] Started GET /blockvolumes/dc2ee508e0a489300de3025b6126876f >[negroni] Completed 200 OK in 1.463448ms >[kubeexec] DEBUG 2018/08/19 12:39:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:42 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3f06586c6e46a53541a8fe1be8f101fa >[negroni] Completed 200 OK in 130.356µs >[negroni] Started GET /queue/3f06586c6e46a53541a8fe1be8f101fa >[negroni] Completed 200 OK in 114.943µs >[kubeexec] DEBUG 2018/08/19 12:39:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_a1093216f64e31232050d160e97ddcbf ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:9d6c5622-907a-48d6-b592-73025be33760", "USERNAME": "9d6c5622-907a-48d6-b592-73025be33760", "PASSWORD": "e398e5b2-beb6-46d1-baa1-5d953f89c5b0", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:39:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:44 asynchttp.go:292: Completed job 3f06586c6e46a53541a8fe1be8f101fa in 2.557783853s >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 12:39:45 Free size 2 is less than the requested block volume size 5 >[heketi] WARNING 2018/08/19 12:39:45 Free size 3 is less than the requested block volume size 5 >[negroni] Completed 202 Accepted in 20.128109ms >[asynchttp] INFO 2018/08/19 12:39:45 asynchttp.go:288: Started job 01332a14449aa4a348f9e633b19ba6b7 >[heketi] INFO 2018/08/19 12:39:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/01332a14449aa4a348f9e633b19ba6b7 >[negroni] Completed 200 OK in 119.409µs >[cmdexec] INFO 2018/08/19 12:39:45 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[cmdexec] INFO 2018/08/19 12:39:45 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 12:39:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3f06586c6e46a53541a8fe1be8f101fa >[negroni] Completed 303 See Other in 121.847µs >[negroni] Started GET /blockvolumes/a1093216f64e31232050d160e97ddcbf >[negroni] Completed 200 OK in 1.562602ms >[kubeexec] DEBUG 2018/08/19 12:39:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 12:39:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:39:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/01332a14449aa4a348f9e633b19ba6b7 >[negroni] Completed 200 OK in 133.427µs >[negroni] Started GET /queue/01332a14449aa4a348f9e633b19ba6b7 >[negroni] Completed 200 OK in 154.848µs >[kubeexec] DEBUG 2018/08/19 12:39:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_81ba05105b6346cea9db718e810486b5 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 5GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:07b34e42-0c63-4c4b-af23-5fe8a6d2b0b6", "USERNAME": "07b34e42-0c63-4c4b-af23-5fe8a6d2b0b6", "PASSWORD": "f1b28703-2d7f-4e57-8626-ed8573a09110", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/01332a14449aa4a348f9e633b19ba6b7 >[negroni] Completed 200 OK in 160.955µs >[heketi] INFO 2018/08/19 12:39:48 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:39:48 asynchttp.go:292: Completed job 01332a14449aa4a348f9e633b19ba6b7 in 3.016514761s >[negroni] Started GET /queue/01332a14449aa4a348f9e633b19ba6b7 >[negroni] Completed 303 See Other in 131.721µs >[negroni] Started GET /blockvolumes/81ba05105b6346cea9db718e810486b5 >[negroni] Completed 200 OK in 2.520321ms >[heketi] INFO 2018/08/19 12:40:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:40:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:40:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:40:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:40:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:40:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 44min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:40:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:40:37 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 2.708485ms >[negroni] Started GET /volumes/00d7e51843ef621811a54f378d5dea3e >[negroni] Completed 200 OK in 1.833957ms >[negroni] Started GET /volumes/3589da219d6536edb00cf7b533976e25 >[negroni] Completed 200 OK in 1.614426ms >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 200 OK in 923.232µs >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 657.4µs >[negroni] Started GET /volumes/8aad36048f6d364c329fa19445a48e92 >[negroni] Completed 200 OK in 530.405µs >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 268.632µs >[negroni] Started GET /volumes/00d7e51843ef621811a54f378d5dea3e >[negroni] Completed 200 OK in 1.226929ms >[negroni] Started GET /volumes/3589da219d6536edb00cf7b533976e25 >[negroni] Completed 200 OK in 674.127µs >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 200 OK in 1.055239ms >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 983.962µs >[negroni] Started GET /volumes/8aad36048f6d364c329fa19445a48e92 >[negroni] Completed 200 OK in 983.253µs >[heketi] INFO 2018/08/19 12:42:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:42:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:42:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:42:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:42:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:42:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 46min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:42:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:42:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:44:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:44:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:44:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:44:36 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:44:36 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:44:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:44:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 48min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:44:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:44:37 Cleaned 0 nodes from health cache >[negroni] Started DELETE /blockvolumes/dc00030249e55f1fd3aecfd06617c194 >[asynchttp] INFO 2018/08/19 12:44:58 asynchttp.go:288: Started job aa4b6e18ca9808f9b31888c47f260a81 >[heketi] INFO 2018/08/19 12:44:58 Started async operation: Delete Block Volume >[negroni] Completed 202 Accepted in 19.835803ms >[negroni] Started GET /queue/aa4b6e18ca9808f9b31888c47f260a81 >[negroni] Completed 200 OK in 146.743µs >[cmdexec] INFO 2018/08/19 12:44:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/aa4b6e18ca9808f9b31888c47f260a81 >[negroni] Completed 200 OK in 172.252µs >[negroni] Started GET /queue/aa4b6e18ca9808f9b31888c47f260a81 >[negroni] Completed 200 OK in 200.904µs >[negroni] Started DELETE /blockvolumes/5efba45d201e343007e3849ad1997aa5 >[negroni] Completed 202 Accepted in 8.648439ms >[asynchttp] INFO 2018/08/19 12:45:00 asynchttp.go:288: Started job ef13e351c2566e41d6abc042837ff585 >[heketi] INFO 2018/08/19 12:45:00 Started async operation: Delete Block Volume >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 200 OK in 82.118µs >[cmdexec] INFO 2018/08/19 12:45:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/aa4b6e18ca9808f9b31888c47f260a81 >[negroni] Completed 200 OK in 164.093µs >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 200 OK in 140.788µs >[kubeexec] DEBUG 2018/08/19 12:45:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_dc00030249e55f1fd3aecfd06617c194 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 12:45:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:45:02 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:02 asynchttp.go:292: Completed job aa4b6e18ca9808f9b31888c47f260a81 in 3.98368842s >[negroni] Started GET /queue/aa4b6e18ca9808f9b31888c47f260a81 >[negroni] Completed 204 No Content in 145.18µs >[negroni] Started DELETE /blockvolumes/dc00030249e55f1fd3aecfd06617c194 >[negroni] Completed 404 Not Found in 3.028118ms >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 200 OK in 111.673µs >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 200 OK in 168.718µs >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 200 OK in 124.222µs >[kubeexec] DEBUG 2018/08/19 12:45:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_5efba45d201e343007e3849ad1997aa5 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:05 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:05 asynchttp.go:292: Completed job ef13e351c2566e41d6abc042837ff585 in 4.991187767s >[negroni] Started GET /queue/ef13e351c2566e41d6abc042837ff585 >[negroni] Completed 204 No Content in 154.465µs >[negroni] Started DELETE /blockvolumes/5efba45d201e343007e3849ad1997aa5 >[negroni] Completed 404 Not Found in 2.270337ms >[negroni] Started DELETE /blockvolumes/ae15f47e58e1afb8f41532c2bc892676 >[negroni] Started DELETE /blockvolumes/734b361391d3cf14150a14f7a0a82632 >[negroni] Completed 202 Accepted in 9.791978ms >[asynchttp] INFO 2018/08/19 12:45:09 asynchttp.go:288: Started job 36239cde05ec247c37cbd344e3f4b0b7 >[heketi] INFO 2018/08/19 12:45:09 Started async operation: Delete Block Volume >[negroni] Started GET /queue/36239cde05ec247c37cbd344e3f4b0b7 >[negroni] Completed 200 OK in 194.171µs >[cmdexec] INFO 2018/08/19 12:45:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Completed 202 Accepted in 11.842625ms >[asynchttp] INFO 2018/08/19 12:45:09 asynchttp.go:288: Started job 8f47a82a1450d9262ff58e5a192e1dcc >[heketi] INFO 2018/08/19 12:45:09 Started async operation: Delete Block Volume >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 134.935µs >[cmdexec] INFO 2018/08/19 12:45:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:45:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] DEBUG 2018/08/19 12:45:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/36239cde05ec247c37cbd344e3f4b0b7 >[negroni] Completed 200 OK in 164.011µs >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 165.58µs >[negroni] Started GET /queue/36239cde05ec247c37cbd344e3f4b0b7 >[negroni] Completed 200 OK in 189.808µs >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 216.583µs >[negroni] Started GET /queue/36239cde05ec247c37cbd344e3f4b0b7 >[negroni] Completed 200 OK in 140.516µs >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 126.061µs >[kubeexec] DEBUG 2018/08/19 12:45:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_ae15f47e58e1afb8f41532c2bc892676 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:12 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:12 asynchttp.go:292: Completed job 36239cde05ec247c37cbd344e3f4b0b7 in 3.155764265s >[negroni] Started GET /queue/36239cde05ec247c37cbd344e3f4b0b7 >[negroni] Completed 204 No Content in 206.206µs >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 119.807µs >[negroni] Started DELETE /blockvolumes/296f9f8049fcd10e5ece2e901bc394e7 >[negroni] Completed 202 Accepted in 15.831787ms >[asynchttp] INFO 2018/08/19 12:45:13 asynchttp.go:288: Started job 71b469beab43af5d5254ca01f8c8bbd4 >[heketi] INFO 2018/08/19 12:45:13 Started async operation: Delete Block Volume >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 171.094µs >[cmdexec] INFO 2018/08/19 12:45:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 204.948µs >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 204.764µs >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 200 OK in 175.331µs >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 84.916µs >[kubeexec] DEBUG 2018/08/19 12:45:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_734b361391d3cf14150a14f7a0a82632 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:16 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:16 asynchttp.go:292: Completed job 8f47a82a1450d9262ff58e5a192e1dcc in 6.193845597s >[kubeexec] DEBUG 2018/08/19 12:45:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8f47a82a1450d9262ff58e5a192e1dcc >[negroni] Completed 204 No Content in 132.075µs >[negroni] Started DELETE /blockvolumes/803ddc4bd13e656f0cd960eaed4b9284 >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 126.203µs >[negroni] Completed 202 Accepted in 15.397884ms >[asynchttp] INFO 2018/08/19 12:45:16 asynchttp.go:288: Started job 05bfbb4a8b45bc40bfaa2520c9577f3d >[heketi] INFO 2018/08/19 12:45:16 Started async operation: Delete Block Volume >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 132.023µs >[cmdexec] INFO 2018/08/19 12:45:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 129.086µs >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 135.089µs >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 200 OK in 179.306µs >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 142.201µs >[kubeexec] DEBUG 2018/08/19 12:45:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_296f9f8049fcd10e5ece2e901bc394e7 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:19 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:19 asynchttp.go:292: Completed job 71b469beab43af5d5254ca01f8c8bbd4 in 5.1678657s >[kubeexec] DEBUG 2018/08/19 12:45:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/71b469beab43af5d5254ca01f8c8bbd4 >[negroni] Completed 204 No Content in 189.133µs >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 87.907µs >[negroni] Started DELETE /blockvolumes/c6fc4619ea4b826924e492375fd1aa98 >[negroni] Completed 202 Accepted in 14.468161ms >[asynchttp] INFO 2018/08/19 12:45:19 asynchttp.go:288: Started job 49c10aeba98a8cc2f957b9128055e61e >[heketi] INFO 2018/08/19 12:45:19 Started async operation: Delete Block Volume >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 191.872µs >[cmdexec] INFO 2018/08/19 12:45:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 159.199µs >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 168.343µs >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 200 OK in 191.408µs >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 123.668µs >[kubeexec] DEBUG 2018/08/19 12:45:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_803ddc4bd13e656f0cd960eaed4b9284 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:22 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:22 asynchttp.go:292: Completed job 05bfbb4a8b45bc40bfaa2520c9577f3d in 5.154358284s >[kubeexec] DEBUG 2018/08/19 12:45:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/05bfbb4a8b45bc40bfaa2520c9577f3d >[negroni] Completed 204 No Content in 236.474µs >[negroni] Started DELETE /blockvolumes/1d0c5f8778c4fb0c4b444c284f300526 >[negroni] Completed 202 Accepted in 11.598597ms >[asynchttp] INFO 2018/08/19 12:45:22 asynchttp.go:288: Started job f72790f23810a4f9a014c00a65f7288f >[heketi] INFO 2018/08/19 12:45:22 Started async operation: Delete Block Volume >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 132.304µs >[cmdexec] INFO 2018/08/19 12:45:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 70.028µs >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 162.525µs >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 122.919µs >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 172.348µs >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 200 OK in 89.353µs >[kubeexec] DEBUG 2018/08/19 12:45:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_c6fc4619ea4b826924e492375fd1aa98 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:25 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:25 asynchttp.go:292: Completed job 49c10aeba98a8cc2f957b9128055e61e in 5.175074745s >[kubeexec] DEBUG 2018/08/19 12:45:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 170.201µs >[negroni] Started GET /queue/49c10aeba98a8cc2f957b9128055e61e >[negroni] Completed 204 No Content in 111.547µs >[negroni] Started DELETE /blockvolumes/c9a3c0de2187199b38a6eebf73d6b19a >[negroni] Completed 202 Accepted in 12.025828ms >[asynchttp] INFO 2018/08/19 12:45:25 asynchttp.go:288: Started job 721c71ea45c54efb1e1ceb91a7ad8782 >[heketi] INFO 2018/08/19 12:45:25 Started async operation: Delete Block Volume >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 200 OK in 166.781µs >[cmdexec] INFO 2018/08/19 12:45:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 210.128µs >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 200 OK in 85.273µs >[kubeexec] DEBUG 2018/08/19 12:45:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_1d0c5f8778c4fb0c4b444c284f300526 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.46.152", "10.70.47.183" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 200 OK in 102.252µs >[heketi] INFO 2018/08/19 12:45:27 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:27 asynchttp.go:292: Completed job f72790f23810a4f9a014c00a65f7288f in 5.011005294s >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 200 OK in 102.607µs >[kubeexec] DEBUG 2018/08/19 12:45:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f72790f23810a4f9a014c00a65f7288f >[negroni] Completed 204 No Content in 203.276µs >[negroni] Started DELETE /blockvolumes/a50ecea6fbf8f8e45e581b261726a0c8 >[negroni] Completed 202 Accepted in 11.76893ms >[asynchttp] INFO 2018/08/19 12:45:28 asynchttp.go:288: Started job 26165912b8a66a834cec9b651529f484 >[heketi] INFO 2018/08/19 12:45:28 Started async operation: Delete Block Volume >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 200 OK in 104.381µs >[cmdexec] INFO 2018/08/19 12:45:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 200 OK in 152.125µs >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 200 OK in 161.384µs >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 200 OK in 141.057µs >[kubeexec] DEBUG 2018/08/19 12:45:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_c9a3c0de2187199b38a6eebf73d6b19a --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:30 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:30 asynchttp.go:292: Completed job 721c71ea45c54efb1e1ceb91a7ad8782 in 4.901958677s >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 200 OK in 137.272µs >[kubeexec] DEBUG 2018/08/19 12:45:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/721c71ea45c54efb1e1ceb91a7ad8782 >[negroni] Completed 204 No Content in 126.377µs >[negroni] Started DELETE /blockvolumes/72541f809a1332bc00dd6123976a2e61 >[negroni] Completed 202 Accepted in 11.89187ms >[asynchttp] INFO 2018/08/19 12:45:30 asynchttp.go:288: Started job afb9c9781bb0283dd046bdc620d774b8 >[heketi] INFO 2018/08/19 12:45:30 Started async operation: Delete Block Volume >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 118.496µs >[cmdexec] INFO 2018/08/19 12:45:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 200 OK in 162.108µs >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 195.645µs >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 200 OK in 116.561µs >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 316.505µs >[kubeexec] DEBUG 2018/08/19 12:45:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_a50ecea6fbf8f8e45e581b261726a0c8 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.46.152", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:33 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:33 asynchttp.go:292: Completed job 26165912b8a66a834cec9b651529f484 in 4.576605824s >[kubeexec] DEBUG 2018/08/19 12:45:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/26165912b8a66a834cec9b651529f484 >[negroni] Completed 204 No Content in 142.1µs >[negroni] Started DELETE /blockvolumes/37db5d4fc7f80928a69639cc37500018 >[negroni] Completed 202 Accepted in 9.636972ms >[asynchttp] INFO 2018/08/19 12:45:33 asynchttp.go:288: Started job db45b32482d7446a1abe46d329fea8b4 >[heketi] INFO 2018/08/19 12:45:33 Started async operation: Delete Block Volume >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 200 OK in 73.415µs >[cmdexec] INFO 2018/08/19 12:45:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 85.41µs >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 200 OK in 120.423µs >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 169.353µs >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 200 OK in 156.609µs >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 200 OK in 105.98µs >[kubeexec] DEBUG 2018/08/19 12:45:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_72541f809a1332bc00dd6123976a2e61 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:36 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:36 asynchttp.go:292: Completed job afb9c9781bb0283dd046bdc620d774b8 in 5.140307815s >[kubeexec] DEBUG 2018/08/19 12:45:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 200 OK in 137.081µs >[negroni] Started GET /queue/afb9c9781bb0283dd046bdc620d774b8 >[negroni] Completed 204 No Content in 180.019µs >[negroni] Started DELETE /blockvolumes/977042bff747c432ba64322466a74cfd >[negroni] Completed 202 Accepted in 10.463203ms >[asynchttp] INFO 2018/08/19 12:45:37 asynchttp.go:288: Started job 4375e069b0651a978233af9070bc7cad >[heketi] INFO 2018/08/19 12:45:37 Started async operation: Delete Block Volume >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 200 OK in 127.471µs >[cmdexec] INFO 2018/08/19 12:45:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 200 OK in 128.294µs >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 200 OK in 133.343µs >[kubeexec] DEBUG 2018/08/19 12:45:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete blkhosting/blockvol_37db5d4fc7f80928a69639cc37500018 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:38 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:38 asynchttp.go:292: Completed job db45b32482d7446a1abe46d329fea8b4 in 4.811615102s >[kubeexec] DEBUG 2018/08/19 12:45:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/db45b32482d7446a1abe46d329fea8b4 >[negroni] Completed 204 No Content in 123.695µs >[negroni] Started DELETE /blockvolumes/0f7c9b41875bce6c6522dda1f94a9ee6 >[negroni] Completed 202 Accepted in 12.056684ms >[asynchttp] INFO 2018/08/19 12:45:38 asynchttp.go:288: Started job 9d413a8488850e81d71b23ba21198616 >[heketi] INFO 2018/08/19 12:45:38 Started async operation: Delete Block Volume >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 200 OK in 137.527µs >[cmdexec] INFO 2018/08/19 12:45:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 200 OK in 201.773µs >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 200 OK in 170.634µs >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 200 OK in 135.068µs >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 200 OK in 167.374µs >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 200 OK in 130.498µs >[kubeexec] DEBUG 2018/08/19 12:45:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_977042bff747c432ba64322466a74cfd --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:41 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:41 asynchttp.go:292: Completed job 4375e069b0651a978233af9070bc7cad in 4.261098522s >[kubeexec] DEBUG 2018/08/19 12:45:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 200 OK in 139.478µs >[negroni] Started GET /queue/4375e069b0651a978233af9070bc7cad >[negroni] Completed 204 No Content in 140.412µs >[negroni] Started DELETE /blockvolumes/16fa08bd873c994ba86ebef232aa11ff >[negroni] Completed 202 Accepted in 11.233277ms >[asynchttp] INFO 2018/08/19 12:45:42 asynchttp.go:288: Started job 68356f55e05e3fa19ee55ab0cd63b761 >[heketi] INFO 2018/08/19 12:45:42 Started async operation: Delete Block Volume >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 200 OK in 149.993µs >[cmdexec] INFO 2018/08/19 12:45:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 200 OK in 222.834µs >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 200 OK in 131.029µs >[kubeexec] DEBUG 2018/08/19 12:45:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_0f7c9b41875bce6c6522dda1f94a9ee6 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:43 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:43 asynchttp.go:292: Completed job 9d413a8488850e81d71b23ba21198616 in 4.746887483s >[kubeexec] DEBUG 2018/08/19 12:45:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9d413a8488850e81d71b23ba21198616 >[negroni] Completed 204 No Content in 144.254µs >[negroni] Started DELETE /blockvolumes/eeb28dde662192a703d255fd2e53d0d7 >[negroni] Completed 202 Accepted in 11.176431ms >[asynchttp] INFO 2018/08/19 12:45:44 asynchttp.go:288: Started job 7a6151b64b5645c9d781908ba084d163 >[heketi] INFO 2018/08/19 12:45:44 Started async operation: Delete Block Volume >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 200 OK in 156.388µs >[cmdexec] INFO 2018/08/19 12:45:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 200 OK in 151.337µs >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 200 OK in 150.964µs >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 200 OK in 202.522µs >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 200 OK in 147.297µs >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 200 OK in 189.397µs >[kubeexec] DEBUG 2018/08/19 12:45:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_16fa08bd873c994ba86ebef232aa11ff --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:46 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:46 asynchttp.go:292: Completed job 68356f55e05e3fa19ee55ab0cd63b761 in 4.140699957s >[kubeexec] DEBUG 2018/08/19 12:45:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 200 OK in 191.356µs >[negroni] Started GET /queue/68356f55e05e3fa19ee55ab0cd63b761 >[negroni] Completed 204 No Content in 171.765µs >[negroni] Started DELETE /blockvolumes/82a1b738e81c902823a1c93b06dd35f9 >[negroni] Completed 202 Accepted in 12.371785ms >[asynchttp] INFO 2018/08/19 12:45:47 asynchttp.go:288: Started job 46deeb86bbbbe5eae1defb9290f74483 >[heketi] INFO 2018/08/19 12:45:47 Started async operation: Delete Block Volume >[negroni] Started GET /queue/46deeb86bbbbe5eae1defb9290f74483 >[negroni] Completed 200 OK in 776.941µs >[cmdexec] INFO 2018/08/19 12:45:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 200 OK in 161.738µs >[negroni] Started GET /queue/46deeb86bbbbe5eae1defb9290f74483 >[negroni] Completed 200 OK in 99.239µs >[kubeexec] DEBUG 2018/08/19 12:45:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_eeb28dde662192a703d255fd2e53d0d7 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:48 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:48 asynchttp.go:292: Completed job 7a6151b64b5645c9d781908ba084d163 in 4.553470345s >[kubeexec] DEBUG 2018/08/19 12:45:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7a6151b64b5645c9d781908ba084d163 >[negroni] Completed 204 No Content in 212.587µs >[negroni] Started DELETE /blockvolumes/0cc4cd96fd5d7b9ff7b76fd03a6eca1a >[negroni] Completed 202 Accepted in 10.193879ms >[asynchttp] INFO 2018/08/19 12:45:49 asynchttp.go:288: Started job 5faa733e5c41722b9524592141167c30 >[heketi] INFO 2018/08/19 12:45:49 Started async operation: Delete Block Volume >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 200 OK in 91.22µs >[cmdexec] INFO 2018/08/19 12:45:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/46deeb86bbbbe5eae1defb9290f74483 >[negroni] Completed 200 OK in 104.792µs >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 200 OK in 244.794µs >[negroni] Started GET /queue/46deeb86bbbbe5eae1defb9290f74483 >[negroni] Completed 200 OK in 113.183µs >[kubeexec] DEBUG 2018/08/19 12:45:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_82a1b738e81c902823a1c93b06dd35f9 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.46.152", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:50 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:50 asynchttp.go:292: Completed job 46deeb86bbbbe5eae1defb9290f74483 in 3.814023278s >[kubeexec] DEBUG 2018/08/19 12:45:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 200 OK in 119.206µs >[negroni] Started GET /queue/46deeb86bbbbe5eae1defb9290f74483 >[negroni] Completed 204 No Content in 129.632µs >[negroni] Started DELETE /blockvolumes/9381f35b40ad3d968df563142bf890aa >[negroni] Completed 202 Accepted in 11.448796ms >[asynchttp] INFO 2018/08/19 12:45:51 asynchttp.go:288: Started job 16427e2abbdc9f336c79b923ffa02758 >[heketi] INFO 2018/08/19 12:45:51 Started async operation: Delete Block Volume >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 200 OK in 137.886µs >[cmdexec] INFO 2018/08/19 12:45:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 200 OK in 122.68µs >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 200 OK in 158.174µs >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 200 OK in 190.894µs >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 200 OK in 117.16µs >[kubeexec] DEBUG 2018/08/19 12:45:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_0cc4cd96fd5d7b9ff7b76fd03a6eca1a --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:53 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:53 asynchttp.go:292: Completed job 5faa733e5c41722b9524592141167c30 in 4.079178355s >[kubeexec] DEBUG 2018/08/19 12:45:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5faa733e5c41722b9524592141167c30 >[negroni] Completed 204 No Content in 184.968µs >[negroni] Started DELETE /blockvolumes/933aaf480a75edaf20c4c0183b213632 >[negroni] Completed 202 Accepted in 10.700025ms >[asynchttp] INFO 2018/08/19 12:45:54 asynchttp.go:288: Started job a6dbf3c87fe1d48f52b4d76f238b6ed5 >[heketi] INFO 2018/08/19 12:45:54 Started async operation: Delete Block Volume >[negroni] Started GET /queue/a6dbf3c87fe1d48f52b4d76f238b6ed5 >[negroni] Completed 200 OK in 90.891µs >[cmdexec] INFO 2018/08/19 12:45:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 200 OK in 200.835µs >[negroni] Started GET /queue/a6dbf3c87fe1d48f52b4d76f238b6ed5 >[negroni] Completed 200 OK in 149.795µs >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 200 OK in 129.075µs >[kubeexec] DEBUG 2018/08/19 12:45:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_9381f35b40ad3d968df563142bf890aa --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.46.152", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:55 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:55 asynchttp.go:292: Completed job 16427e2abbdc9f336c79b923ffa02758 in 4.752771195s >[kubeexec] DEBUG 2018/08/19 12:45:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a6dbf3c87fe1d48f52b4d76f238b6ed5 >[negroni] Completed 200 OK in 172.095µs >[negroni] Started GET /queue/16427e2abbdc9f336c79b923ffa02758 >[negroni] Completed 204 No Content in 204.19µs >[negroni] Started DELETE /blockvolumes/7e79dab7aed560396bc8630a86bfe403 >[negroni] Completed 202 Accepted in 12.076836ms >[asynchttp] INFO 2018/08/19 12:45:56 asynchttp.go:288: Started job 8759ad689a8a7419dfe791455613ab10 >[heketi] INFO 2018/08/19 12:45:56 Started async operation: Delete Block Volume >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 200 OK in 100.567µs >[cmdexec] INFO 2018/08/19 12:45:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a6dbf3c87fe1d48f52b4d76f238b6ed5 >[negroni] Completed 200 OK in 162.659µs >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 200 OK in 187.55µs >[kubeexec] DEBUG 2018/08/19 12:45:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_933aaf480a75edaf20c4c0183b213632 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:45:57 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:45:57 asynchttp.go:292: Completed job a6dbf3c87fe1d48f52b4d76f238b6ed5 in 3.889721746s >[kubeexec] DEBUG 2018/08/19 12:45:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a6dbf3c87fe1d48f52b4d76f238b6ed5 >[negroni] Completed 204 No Content in 128.939µs >[negroni] Started DELETE /blockvolumes/7b7e896fa83a35604f8d5ae607b49cd8 >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 200 OK in 148.656µs >[negroni] Completed 202 Accepted in 261.193577ms >[asynchttp] INFO 2018/08/19 12:45:58 asynchttp.go:288: Started job 4901cb708b4b5c42830f4ee2a0715ef5 >[heketi] INFO 2018/08/19 12:45:58 Started async operation: Delete Block Volume >[negroni] Started GET /queue/4901cb708b4b5c42830f4ee2a0715ef5 >[negroni] Completed 200 OK in 731.188µs >[cmdexec] INFO 2018/08/19 12:45:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 200 OK in 226.279µs >[negroni] Started GET /queue/4901cb708b4b5c42830f4ee2a0715ef5 >[negroni] Completed 200 OK in 138.063µs >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 200 OK in 170.791µs >[kubeexec] DEBUG 2018/08/19 12:46:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_7e79dab7aed560396bc8630a86bfe403 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.46.152", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:00 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:00 asynchttp.go:292: Completed job 8759ad689a8a7419dfe791455613ab10 in 4.11332706s >[negroni] Started GET /queue/4901cb708b4b5c42830f4ee2a0715ef5 >[negroni] Completed 200 OK in 135.588µs >[kubeexec] DEBUG 2018/08/19 12:46:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8759ad689a8a7419dfe791455613ab10 >[negroni] Completed 204 No Content in 123.139µs >[negroni] Started DELETE /blockvolumes/3ba48c0b9bc39e3c4cfc1ee4e667c9a6 >[negroni] Completed 202 Accepted in 9.400503ms >[asynchttp] INFO 2018/08/19 12:46:01 asynchttp.go:288: Started job 9b87e498dcdd1ed14eb41994fa90f51d >[heketi] INFO 2018/08/19 12:46:01 Started async operation: Delete Block Volume >[negroni] Started GET /queue/9b87e498dcdd1ed14eb41994fa90f51d >[negroni] Completed 200 OK in 92.296µs >[cmdexec] INFO 2018/08/19 12:46:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4901cb708b4b5c42830f4ee2a0715ef5 >[negroni] Completed 200 OK in 216.529µs >[negroni] Started GET /queue/9b87e498dcdd1ed14eb41994fa90f51d >[negroni] Completed 200 OK in 188.586µs >[kubeexec] DEBUG 2018/08/19 12:46:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_7b7e896fa83a35604f8d5ae607b49cd8 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:02 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:02 asynchttp.go:292: Completed job 4901cb708b4b5c42830f4ee2a0715ef5 in 3.886612598s >[kubeexec] DEBUG 2018/08/19 12:46:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4901cb708b4b5c42830f4ee2a0715ef5 >[negroni] Completed 204 No Content in 161.116µs >[negroni] Started DELETE /blockvolumes/ba51b81fbb61e6240b61fd205cb357c3 >[negroni] Completed 202 Accepted in 17.471754ms >[asynchttp] INFO 2018/08/19 12:46:02 asynchttp.go:288: Started job 286b28241d9b20815e9b778c6a6341c2 >[heketi] INFO 2018/08/19 12:46:02 Started async operation: Delete Block Volume >[negroni] Started GET /queue/286b28241d9b20815e9b778c6a6341c2 >[negroni] Completed 200 OK in 128.249µs >[cmdexec] INFO 2018/08/19 12:46:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9b87e498dcdd1ed14eb41994fa90f51d >[negroni] Completed 200 OK in 154.1µs >[negroni] Started GET /queue/286b28241d9b20815e9b778c6a6341c2 >[negroni] Completed 200 OK in 151.417µs >[negroni] Started GET /queue/9b87e498dcdd1ed14eb41994fa90f51d >[negroni] Completed 200 OK in 160.304µs >[kubeexec] DEBUG 2018/08/19 12:46:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_45a4ba9ee3f5a61061b35d95ae58adc8/blockvol_3ba48c0b9bc39e3c4cfc1ee4e667c9a6 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:04 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:04 asynchttp.go:292: Completed job 9b87e498dcdd1ed14eb41994fa90f51d in 3.04814257s >[kubeexec] DEBUG 2018/08/19 12:46:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/286b28241d9b20815e9b778c6a6341c2 >[negroni] Completed 200 OK in 137.995µs >[negroni] Started GET /queue/9b87e498dcdd1ed14eb41994fa90f51d >[negroni] Completed 204 No Content in 125.366µs >[negroni] Started DELETE /blockvolumes/97cf559b4833e868b22204919b861aff >[negroni] Completed 202 Accepted in 11.04834ms >[asynchttp] INFO 2018/08/19 12:46:05 asynchttp.go:288: Started job d71c1695a58b228e87354167a9e662db >[heketi] INFO 2018/08/19 12:46:05 Started async operation: Delete Block Volume >[negroni] Started GET /queue/d71c1695a58b228e87354167a9e662db >[negroni] Completed 200 OK in 81.978µs >[cmdexec] INFO 2018/08/19 12:46:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/286b28241d9b20815e9b778c6a6341c2 >[negroni] Completed 200 OK in 142.238µs >[kubeexec] DEBUG 2018/08/19 12:46:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_ba51b81fbb61e6240b61fd205cb357c3 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:06 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:06 asynchttp.go:292: Completed job 286b28241d9b20815e9b778c6a6341c2 in 3.742565902s >[negroni] Started GET /queue/d71c1695a58b228e87354167a9e662db >[negroni] Completed 200 OK in 125.556µs >[kubeexec] DEBUG 2018/08/19 12:46:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/286b28241d9b20815e9b778c6a6341c2 >[negroni] Completed 204 No Content in 123.878µs >[negroni] Started DELETE /blockvolumes/badbf77121230eefaf245e08553ab5a5 >[negroni] Completed 202 Accepted in 9.818001ms >[asynchttp] INFO 2018/08/19 12:46:06 asynchttp.go:288: Started job 90608f9b994ea3cf72fe894af62755cb >[heketi] INFO 2018/08/19 12:46:06 Started async operation: Delete Block Volume >[negroni] Started GET /queue/90608f9b994ea3cf72fe894af62755cb >[negroni] Completed 200 OK in 115.25µs >[cmdexec] INFO 2018/08/19 12:46:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/d71c1695a58b228e87354167a9e662db >[negroni] Completed 200 OK in 158.507µs >[negroni] Started GET /queue/90608f9b994ea3cf72fe894af62755cb >[negroni] Completed 200 OK in 122.244µs >[kubeexec] DEBUG 2018/08/19 12:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_97cf559b4833e868b22204919b861aff --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 12:46:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:46:08 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:08 asynchttp.go:292: Completed job d71c1695a58b228e87354167a9e662db in 2.935729999s >[negroni] Started GET /queue/d71c1695a58b228e87354167a9e662db >[negroni] Completed 204 No Content in 124.331µs >[negroni] Started DELETE /blockvolumes/dc2ee508e0a489300de3025b6126876f >[negroni] Completed 202 Accepted in 11.743528ms >[asynchttp] INFO 2018/08/19 12:46:08 asynchttp.go:288: Started job ac2d9d91e459e9a42982cc359b235380 >[heketi] INFO 2018/08/19 12:46:08 Started async operation: Delete Block Volume >[negroni] Started GET /queue/ac2d9d91e459e9a42982cc359b235380 >[negroni] Completed 200 OK in 151.05µs >[cmdexec] INFO 2018/08/19 12:46:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/90608f9b994ea3cf72fe894af62755cb >[negroni] Completed 200 OK in 158.313µs >[negroni] Started GET /queue/ac2d9d91e459e9a42982cc359b235380 >[negroni] Completed 200 OK in 208.511µs >[negroni] Started GET /queue/90608f9b994ea3cf72fe894af62755cb >[negroni] Completed 200 OK in 160.553µs >[kubeexec] DEBUG 2018/08/19 12:46:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_badbf77121230eefaf245e08553ab5a5 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:09 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:09 asynchttp.go:292: Completed job 90608f9b994ea3cf72fe894af62755cb in 3.369517471s >[kubeexec] DEBUG 2018/08/19 12:46:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ac2d9d91e459e9a42982cc359b235380 >[negroni] Completed 200 OK in 133.418µs >[negroni] Started GET /queue/90608f9b994ea3cf72fe894af62755cb >[negroni] Completed 204 No Content in 154.928µs >[negroni] Started DELETE /blockvolumes/a1093216f64e31232050d160e97ddcbf >[negroni] Completed 202 Accepted in 9.647959ms >[asynchttp] INFO 2018/08/19 12:46:10 asynchttp.go:288: Started job 59de78699abb4e653e3f09d746bff351 >[heketi] INFO 2018/08/19 12:46:10 Started async operation: Delete Block Volume >[negroni] Started GET /queue/59de78699abb4e653e3f09d746bff351 >[negroni] Completed 200 OK in 111.962µs >[cmdexec] INFO 2018/08/19 12:46:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ac2d9d91e459e9a42982cc359b235380 >[negroni] Completed 200 OK in 579.97µs >[negroni] Started GET /queue/59de78699abb4e653e3f09d746bff351 >[negroni] Completed 200 OK in 159.593µs >[kubeexec] DEBUG 2018/08/19 12:46:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_dc2ee508e0a489300de3025b6126876f --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.46.152", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:11 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:11 asynchttp.go:292: Completed job ac2d9d91e459e9a42982cc359b235380 in 3.250963398s >[kubeexec] DEBUG 2018/08/19 12:46:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ac2d9d91e459e9a42982cc359b235380 >[negroni] Completed 204 No Content in 125.611µs >[negroni] Started DELETE /blockvolumes/81ba05105b6346cea9db718e810486b5 >[negroni] Completed 202 Accepted in 53.714972ms >[asynchttp] INFO 2018/08/19 12:46:12 asynchttp.go:288: Started job eb52012d437f133b4039facb135a77ee >[heketi] INFO 2018/08/19 12:46:12 Started async operation: Delete Block Volume >[negroni] Started GET /queue/eb52012d437f133b4039facb135a77ee >[negroni] Completed 200 OK in 112.497µs >[cmdexec] INFO 2018/08/19 12:46:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/59de78699abb4e653e3f09d746bff351 >[negroni] Completed 200 OK in 192.704µs >[kubeexec] DEBUG 2018/08/19 12:46:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_a1093216f64e31232050d160e97ddcbf --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:13 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:13 asynchttp.go:292: Completed job 59de78699abb4e653e3f09d746bff351 in 2.740352943s >[kubeexec] DEBUG 2018/08/19 12:46:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/eb52012d437f133b4039facb135a77ee >[negroni] Completed 200 OK in 95.901µs >[negroni] Started GET /queue/59de78699abb4e653e3f09d746bff351 >[negroni] Completed 204 No Content in 148.115µs >[negroni] Started DELETE /blockvolumes/d02ba85022744725d25492913f121e0c >[negroni] Completed 202 Accepted in 11.724572ms >[asynchttp] INFO 2018/08/19 12:46:13 asynchttp.go:288: Started job 79e1be13a77c5d4247447d991377d1be >[heketi] INFO 2018/08/19 12:46:13 Started async operation: Delete Block Volume >[negroni] Started GET /queue/79e1be13a77c5d4247447d991377d1be >[negroni] Completed 200 OK in 281.859µs >[cmdexec] INFO 2018/08/19 12:46:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/eb52012d437f133b4039facb135a77ee >[negroni] Completed 200 OK in 117.605µs >[negroni] Started GET /queue/79e1be13a77c5d4247447d991377d1be >[negroni] Completed 200 OK in 207.653µs >[kubeexec] DEBUG 2018/08/19 12:46:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_81ba05105b6346cea9db718e810486b5 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:14 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:14 asynchttp.go:292: Completed job eb52012d437f133b4039facb135a77ee in 2.4843131s >[kubeexec] DEBUG 2018/08/19 12:46:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/eb52012d437f133b4039facb135a77ee >[negroni] Completed 204 No Content in 130.476µs >[negroni] Started DELETE /blockvolumes/f7c458a082c030ea6dee3f03f7f00e63 >[negroni] Completed 202 Accepted in 11.759353ms >[asynchttp] INFO 2018/08/19 12:46:15 asynchttp.go:288: Started job 20dafd52a6c5aa9befd077848174f0e8 >[heketi] INFO 2018/08/19 12:46:15 Started async operation: Delete Block Volume >[negroni] Started GET /queue/20dafd52a6c5aa9befd077848174f0e8 >[negroni] Completed 200 OK in 97.104µs >[cmdexec] INFO 2018/08/19 12:46:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/79e1be13a77c5d4247447d991377d1be >[negroni] Completed 200 OK in 184.416µs >[kubeexec] DEBUG 2018/08/19 12:46:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d02ba85022744725d25492913f121e0c --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:16 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:16 asynchttp.go:292: Completed job 79e1be13a77c5d4247447d991377d1be in 2.763215947s >[negroni] Started GET /queue/20dafd52a6c5aa9befd077848174f0e8 >[negroni] Completed 200 OK in 184.459µs >[kubeexec] DEBUG 2018/08/19 12:46:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/79e1be13a77c5d4247447d991377d1be >[negroni] Completed 204 No Content in 146.762µs >[negroni] Started DELETE /blockvolumes/0a2a601086be6888c9fd939a8eb4be54 >[negroni] Completed 202 Accepted in 9.98362ms >[asynchttp] INFO 2018/08/19 12:46:16 asynchttp.go:288: Started job 4b00c5d53ddfcc43676e79f7e768b9a9 >[heketi] INFO 2018/08/19 12:46:16 Started async operation: Delete Block Volume >[negroni] Started GET /queue/4b00c5d53ddfcc43676e79f7e768b9a9 >[negroni] Completed 200 OK in 111.298µs >[cmdexec] INFO 2018/08/19 12:46:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/20dafd52a6c5aa9befd077848174f0e8 >[negroni] Completed 200 OK in 123.78µs >[negroni] Started GET /queue/4b00c5d53ddfcc43676e79f7e768b9a9 >[negroni] Completed 200 OK in 132.041µs >[kubeexec] DEBUG 2018/08/19 12:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f7c458a082c030ea6dee3f03f7f00e63 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:17 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:17 asynchttp.go:292: Completed job 20dafd52a6c5aa9befd077848174f0e8 in 2.363764788s >[kubeexec] DEBUG 2018/08/19 12:46:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/20dafd52a6c5aa9befd077848174f0e8 >[negroni] Completed 204 No Content in 221.201µs >[negroni] Started DELETE /blockvolumes/afe51517f2a88bcde2d354786fcd6dcd >[negroni] Completed 202 Accepted in 10.868703ms >[asynchttp] INFO 2018/08/19 12:46:18 asynchttp.go:288: Started job 3104ea3b9be1679b1a5bc59a638335ab >[heketi] INFO 2018/08/19 12:46:18 Started async operation: Delete Block Volume >[negroni] Started GET /queue/3104ea3b9be1679b1a5bc59a638335ab >[negroni] Completed 200 OK in 101.305µs >[cmdexec] INFO 2018/08/19 12:46:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4b00c5d53ddfcc43676e79f7e768b9a9 >[negroni] Completed 200 OK in 169.058µs >[kubeexec] DEBUG 2018/08/19 12:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_0a2a601086be6888c9fd939a8eb4be54 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:19 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:19 asynchttp.go:292: Completed job 4b00c5d53ddfcc43676e79f7e768b9a9 in 2.52637653s >[kubeexec] DEBUG 2018/08/19 12:46:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3104ea3b9be1679b1a5bc59a638335ab >[negroni] Completed 200 OK in 179.182µs >[negroni] Started GET /queue/4b00c5d53ddfcc43676e79f7e768b9a9 >[negroni] Completed 204 No Content in 190.36µs >[negroni] Started DELETE /blockvolumes/d6d094d080681a88eb8a463df75913a5 >[negroni] Completed 202 Accepted in 10.678321ms >[asynchttp] INFO 2018/08/19 12:46:19 asynchttp.go:288: Started job 305922311975197fab0a8f22c11ab603 >[heketi] INFO 2018/08/19 12:46:19 Started async operation: Delete Block Volume >[negroni] Started GET /queue/305922311975197fab0a8f22c11ab603 >[negroni] Completed 200 OK in 75.412µs >[cmdexec] INFO 2018/08/19 12:46:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3104ea3b9be1679b1a5bc59a638335ab >[negroni] Completed 200 OK in 153.603µs >[kubeexec] DEBUG 2018/08/19 12:46:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_afe51517f2a88bcde2d354786fcd6dcd --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:20 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:20 asynchttp.go:292: Completed job 3104ea3b9be1679b1a5bc59a638335ab in 2.022667381s >[kubeexec] DEBUG 2018/08/19 12:46:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/305922311975197fab0a8f22c11ab603 >[negroni] Completed 200 OK in 141.237µs >[negroni] Started GET /queue/3104ea3b9be1679b1a5bc59a638335ab >[negroni] Completed 204 No Content in 131.082µs >[negroni] Started DELETE /blockvolumes/fe94ebd58bfd864d039f96b9f6849940 >[negroni] Completed 202 Accepted in 9.351344ms >[asynchttp] INFO 2018/08/19 12:46:21 asynchttp.go:288: Started job 0497355c4212b4b13cecbb60b54d789f >[heketi] INFO 2018/08/19 12:46:21 Started async operation: Delete Block Volume >[negroni] Started GET /queue/0497355c4212b4b13cecbb60b54d789f >[negroni] Completed 200 OK in 96.137µs >[cmdexec] INFO 2018/08/19 12:46:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/305922311975197fab0a8f22c11ab603 >[negroni] Completed 200 OK in 121.819µs >[kubeexec] DEBUG 2018/08/19 12:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d6d094d080681a88eb8a463df75913a5 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:21 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:21 asynchttp.go:292: Completed job 305922311975197fab0a8f22c11ab603 in 2.120582249s >[kubeexec] DEBUG 2018/08/19 12:46:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0497355c4212b4b13cecbb60b54d789f >[negroni] Completed 200 OK in 166.107µs >[negroni] Started GET /queue/305922311975197fab0a8f22c11ab603 >[negroni] Completed 204 No Content in 175.086µs >[negroni] Started DELETE /blockvolumes/41e2f7bdf689a4483fdc616b625c5f70 >[negroni] Completed 202 Accepted in 11.907544ms >[asynchttp] INFO 2018/08/19 12:46:22 asynchttp.go:288: Started job f8cda51f0ec084e17c792d840fe47ff2 >[heketi] INFO 2018/08/19 12:46:22 Started async operation: Delete Block Volume >[negroni] Started GET /queue/f8cda51f0ec084e17c792d840fe47ff2 >[negroni] Completed 200 OK in 137.973µs >[cmdexec] INFO 2018/08/19 12:46:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_fe94ebd58bfd864d039f96b9f6849940 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.46.152", "10.70.47.183" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:22 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:22 asynchttp.go:292: Completed job 0497355c4212b4b13cecbb60b54d789f in 1.473768381s >[kubeexec] DEBUG 2018/08/19 12:46:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0497355c4212b4b13cecbb60b54d789f >[negroni] Completed 204 No Content in 128.791µs >[negroni] Started DELETE /blockvolumes/50d9ab52accb141c90d87a2e76fa2b10 >[negroni] Completed 202 Accepted in 13.253959ms >[asynchttp] INFO 2018/08/19 12:46:23 asynchttp.go:288: Started job 69eac122af25c4e8deaeb9d0eebb876b >[heketi] INFO 2018/08/19 12:46:23 Started async operation: Delete Block Volume >[negroni] Started GET /queue/69eac122af25c4e8deaeb9d0eebb876b >[negroni] Completed 200 OK in 85.841µs >[cmdexec] INFO 2018/08/19 12:46:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f8cda51f0ec084e17c792d840fe47ff2 >[negroni] Completed 200 OK in 119.558µs >[kubeexec] DEBUG 2018/08/19 12:46:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_41e2f7bdf689a4483fdc616b625c5f70 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.46.152", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:24 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:24 asynchttp.go:292: Completed job f8cda51f0ec084e17c792d840fe47ff2 in 1.516334359s >[kubeexec] DEBUG 2018/08/19 12:46:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69eac122af25c4e8deaeb9d0eebb876b >[negroni] Completed 200 OK in 129.559µs >[negroni] Started GET /queue/f8cda51f0ec084e17c792d840fe47ff2 >[negroni] Completed 204 No Content in 125.041µs >[negroni] Started DELETE /blockvolumes/d78ae8c6fdbf17f6f97bdafa3aa8f124 >[negroni] Completed 202 Accepted in 12.756768ms >[asynchttp] INFO 2018/08/19 12:46:24 asynchttp.go:288: Started job 4443d09c429dba12bc9e34819dd04a26 >[heketi] INFO 2018/08/19 12:46:24 Started async operation: Delete Block Volume >[negroni] Started GET /queue/4443d09c429dba12bc9e34819dd04a26 >[negroni] Completed 200 OK in 97.663µs >[cmdexec] INFO 2018/08/19 12:46:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_50d9ab52accb141c90d87a2e76fa2b10 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.46.152", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:25 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:25 asynchttp.go:292: Completed job 69eac122af25c4e8deaeb9d0eebb876b in 1.715682497s >[kubeexec] DEBUG 2018/08/19 12:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69eac122af25c4e8deaeb9d0eebb876b >[negroni] Completed 204 No Content in 114.483µs >[negroni] Started DELETE /blockvolumes/fb12b37f20dd08a9e0813d1195bf04bb >[negroni] Completed 202 Accepted in 11.468128ms >[asynchttp] INFO 2018/08/19 12:46:25 asynchttp.go:288: Started job 69ea73b0195255ce8fe2a551d781f9cf >[heketi] INFO 2018/08/19 12:46:25 Started async operation: Delete Block Volume >[negroni] Started GET /queue/69ea73b0195255ce8fe2a551d781f9cf >[negroni] Completed 200 OK in 132.619µs >[cmdexec] INFO 2018/08/19 12:46:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4443d09c429dba12bc9e34819dd04a26 >[negroni] Completed 200 OK in 137.117µs >[kubeexec] DEBUG 2018/08/19 12:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d78ae8c6fdbf17f6f97bdafa3aa8f124 --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:26 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:26 asynchttp.go:292: Completed job 4443d09c429dba12bc9e34819dd04a26 in 1.571429568s >[kubeexec] DEBUG 2018/08/19 12:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69ea73b0195255ce8fe2a551d781f9cf >[negroni] Completed 200 OK in 133.44µs >[negroni] Started GET /queue/4443d09c429dba12bc9e34819dd04a26 >[negroni] Completed 204 No Content in 185.385µs >[negroni] Started DELETE /blockvolumes/b8ba116091f1049a3986ebd85f59d0dd >[negroni] Completed 202 Accepted in 9.839542ms >[asynchttp] INFO 2018/08/19 12:46:26 asynchttp.go:288: Started job 3ff6e910d130abd6ba99513afce9242c >[heketi] INFO 2018/08/19 12:46:26 Started async operation: Delete Block Volume >[negroni] Started GET /queue/3ff6e910d130abd6ba99513afce9242c >[negroni] Completed 200 OK in 158.121µs >[cmdexec] INFO 2018/08/19 12:46:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_fb12b37f20dd08a9e0813d1195bf04bb --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:27 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:27 asynchttp.go:292: Completed job 69ea73b0195255ce8fe2a551d781f9cf in 1.669996521s >[kubeexec] DEBUG 2018/08/19 12:46:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69ea73b0195255ce8fe2a551d781f9cf >[negroni] Completed 204 No Content in 122.801µs >[negroni] Started DELETE /blockvolumes/31c27994f63ef3d98cc077c3e21428e9 >[negroni] Completed 202 Accepted in 10.069857ms >[asynchttp] INFO 2018/08/19 12:46:27 asynchttp.go:288: Started job 01b764cb5bb8c1e259adefc10c231da5 >[heketi] INFO 2018/08/19 12:46:27 Started async operation: Delete Block Volume >[negroni] Started GET /queue/01b764cb5bb8c1e259adefc10c231da5 >[negroni] Completed 200 OK in 187.283µs >[cmdexec] INFO 2018/08/19 12:46:27 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3ff6e910d130abd6ba99513afce9242c >[negroni] Completed 200 OK in 209.325µs >[kubeexec] DEBUG 2018/08/19 12:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_b8ba116091f1049a3986ebd85f59d0dd --json >Result: { "SUCCESSFUL ON": [ "10.70.47.183", "10.70.47.54", "10.70.46.152" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 12:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:46:28 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:28 asynchttp.go:292: Completed job 3ff6e910d130abd6ba99513afce9242c in 1.612894659s >[negroni] Started GET /queue/01b764cb5bb8c1e259adefc10c231da5 >[negroni] Completed 200 OK in 176.341µs >[negroni] Started GET /queue/3ff6e910d130abd6ba99513afce9242c >[negroni] Completed 204 No Content in 259.548µs >[negroni] Started DELETE /blockvolumes/c75e354b772e6037920e02b5abc103ea >[negroni] Completed 202 Accepted in 11.596156ms >[asynchttp] INFO 2018/08/19 12:46:28 asynchttp.go:288: Started job 743cfff7a3ea275f89e6aa3189f4b0ca >[heketi] INFO 2018/08/19 12:46:28 Started async operation: Delete Block Volume >[negroni] Started GET /queue/743cfff7a3ea275f89e6aa3189f4b0ca >[negroni] Completed 200 OK in 125.38µs >[cmdexec] INFO 2018/08/19 12:46:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_31c27994f63ef3d98cc077c3e21428e9 --json >Result: { "SUCCESSFUL ON": [ "10.70.46.152", "10.70.47.183", "10.70.47.54" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:28 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:28 asynchttp.go:292: Completed job 01b764cb5bb8c1e259adefc10c231da5 in 1.379342755s >[kubeexec] DEBUG 2018/08/19 12:46:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/01b764cb5bb8c1e259adefc10c231da5 >[negroni] Completed 204 No Content in 162.032µs >[negroni] Started GET /queue/743cfff7a3ea275f89e6aa3189f4b0ca >[negroni] Completed 200 OK in 163.638µs >[kubeexec] DEBUG 2018/08/19 12:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_c75e354b772e6037920e02b5abc103ea --json >Result: { "SUCCESSFUL ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 12:46:29 Delete Block Volume succeeded >[asynchttp] INFO 2018/08/19 12:46:29 asynchttp.go:292: Completed job 743cfff7a3ea275f89e6aa3189f4b0ca in 1.066286556s >[negroni] Started GET /queue/743cfff7a3ea275f89e6aa3189f4b0ca >[negroni] Completed 204 No Content in 128.842µs >[negroni] Started DELETE /blockvolumes/c75e354b772e6037920e02b5abc103ea >[negroni] Completed 404 Not Found in 1.725061ms >[heketi] INFO 2018/08/19 12:46:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:46:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:46:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:46:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:46:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:46:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 50min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:46:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:46:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:48:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:48:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:48:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:48:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:48:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:48:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 52min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:48:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:48:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:50:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:50:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:50:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:50:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:50:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:50:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 54min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:50:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:50:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:52:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:52:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:52:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:52:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:52:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:52:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:52:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 56min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:52:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:52:37 Cleaned 0 nodes from health cache >[negroni] Started GET /volumes >[negroni] Completed 200 OK in 2.036371ms >[negroni] Started GET /volumes/00d7e51843ef621811a54f378d5dea3e >[negroni] Completed 200 OK in 2.542413ms >[negroni] Started GET /volumes/3589da219d6536edb00cf7b533976e25 >[negroni] Completed 200 OK in 1.065008ms >[negroni] Started GET /volumes/45a4ba9ee3f5a61061b35d95ae58adc8 >[negroni] Completed 200 OK in 1.703184ms >[negroni] Started GET /volumes/80ff41fdbb475c0fa055de08db90ab4f >[negroni] Completed 200 OK in 1.002942ms >[negroni] Started GET /volumes/8aad36048f6d364c329fa19445a48e92 >[negroni] Completed 200 OK in 1.466221ms >[negroni] Started GET /blockvolumes >[negroni] Completed 200 OK in 257.385µs >[negroni] Started GET /blockvolumes/d0b08f33621cbf7a7900873eee8dc0a9 >[negroni] Completed 200 OK in 299.755µs >[heketi] INFO 2018/08/19 12:54:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:54:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:54:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:54:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:54:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:54:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:54:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 58min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:54:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:54:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 12:56:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:56:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ7056 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:56:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:56:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ6697 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:56:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:56:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:56:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 0min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ6716 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:56:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:56:37 Cleaned 0 nodes from health cache >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:16 Allocating brick set #0 >[negroni] Completed 202 Accepted in 48.892453ms >[asynchttp] INFO 2018/08/19 12:58:16 asynchttp.go:288: Started job a60cecc2ac0a55a3d6a45bedd61db9c0 >[heketi] INFO 2018/08/19 12:58:16 Started async operation: Create Volume >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 123.431µs >[heketi] INFO 2018/08/19 12:58:16 Creating brick 5363271fa12ac2979f057f463927a91a >[heketi] INFO 2018/08/19 12:58:16 Creating brick 7d13e6899ea05f456177c644bfd67224 >[heketi] INFO 2018/08/19 12:58:16 Creating brick 6c1ce487d6ccf842287e2cabebc1c8f3 >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a >Result: >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3 >Result: >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:17 Allocating brick set #0 >[negroni] Completed 202 Accepted in 23.178892ms >[asynchttp] INFO 2018/08/19 12:58:17 asynchttp.go:288: Started job 0e788e9dc8c7f27b4ed862a8b7bdee79 >[heketi] INFO 2018/08/19 12:58:17 Started async operation: Create Volume >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 169.084µs >[heketi] INFO 2018/08/19 12:58:17 Creating brick 58ca7062c8bd0c92fb4fa224d2beab6c >[heketi] INFO 2018/08/19 12:58:17 Creating brick 46beb54fc781fdfcd258b75a217adbe1 >[heketi] INFO 2018/08/19 12:58:17 Creating brick c5b42219b2fa5bd950b7368e915d11bb >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_6c1ce487d6ccf842287e2cabebc1c8f3 --virtualsize 1048576K --name brick_6c1ce487d6ccf842287e2cabebc1c8f3 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_6c1ce487d6ccf842287e2cabebc1c8f3" created. >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_5363271fa12ac2979f057f463927a91a --virtualsize 1048576K --name brick_5363271fa12ac2979f057f463927a91a >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_5363271fa12ac2979f057f463927a91a" created. >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_c2e0193ff752a03a5bad1305dfc3c7c5 --virtualsize 1048576K --name brick_7d13e6899ea05f456177c644bfd67224 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_7d13e6899ea05f456177c644bfd67224" created. >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:17 Allocating brick set #0 >[negroni] Completed 202 Accepted in 149.075435ms >[asynchttp] INFO 2018/08/19 12:58:17 asynchttp.go:288: Started job dcc0e27239094affd743114ff7124193 >[heketi] INFO 2018/08/19 12:58:17 Started async operation: Create Volume >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_7d13e6899ea05f456177c644bfd67224 >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_7d13e6899ea05f456177c644bfd67224 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 117.917µs >[heketi] INFO 2018/08/19 12:58:17 Creating brick f5124a847a7f09bce7401fdfd395280f >[heketi] INFO 2018/08/19 12:58:17 Creating brick 26a67e0af8755d7a1b5d94b44addf1ab >[heketi] INFO 2018/08/19 12:58:17 Creating brick 77c490cd008d35f5a4ae9fc53242e1a5 >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_6c1ce487d6ccf842287e2cabebc1c8f3 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_6c1ce487d6ccf842287e2cabebc1c8f3 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 147.906µs >[kubeexec] DEBUG 2018/08/19 12:58:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_7d13e6899ea05f456177c644bfd67224 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5363271fa12ac2979f057f463927a91a >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5363271fa12ac2979f057f463927a91a isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_6c1ce487d6ccf842287e2cabebc1c8f3 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:18 Allocating brick set #0 >[negroni] Completed 202 Accepted in 18.049983ms >[asynchttp] INFO 2018/08/19 12:58:18 asynchttp.go:288: Started job 86879cb72a8a124f762947bebb132c93 >[heketi] INFO 2018/08/19 12:58:18 Started async operation: Create Volume >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 159.44µs >[heketi] INFO 2018/08/19 12:58:18 Creating brick 49f8272024aac926535db90baff49a6e >[heketi] INFO 2018/08/19 12:58:18 Creating brick 5a268d74742c57864d5dc7db3a16c2ee >[heketi] INFO 2018/08/19 12:58:18 Creating brick 106bd6171c93e30bd17842688b5cc99e >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5363271fa12ac2979f057f463927a91a /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_7d13e6899ea05f456177c644bfd67224 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_6c1ce487d6ccf842287e2cabebc1c8f3 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5363271fa12ac2979f057f463927a91a /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 160.224µs >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2001 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2001 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224/brick >Result: >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:18 Allocating brick set #0 >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2001 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a/brick >Result: >[negroni] Completed 202 Accepted in 78.934428ms >[asynchttp] INFO 2018/08/19 12:58:18 asynchttp.go:288: Started job 06bc4d4c3e6cf2cc2339a64c9d216102 >[heketi] INFO 2018/08/19 12:58:18 Started async operation: Create Volume >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 123.68µs >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1 >Result: >[heketi] INFO 2018/08/19 12:58:18 Creating brick 113d866e76ba6fecdb460f41734f57d5 >[heketi] INFO 2018/08/19 12:58:18 Creating brick fb7cb684a9a164563eae1fb333da5088 >[heketi] INFO 2018/08/19 12:58:18 Creating brick 67a0f677207005db8d0e662dcc651154 >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:18 Creating volume vol_6220bc2df0dad69f3f16527d535ebb64 replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c >Result: >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 116.88µs >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:18 Allocating brick set #0 >[negroni] Completed 202 Accepted in 17.926647ms >[asynchttp] INFO 2018/08/19 12:58:18 asynchttp.go:288: Started job e1294465722eb6f8c1c2842bde328ecf >[heketi] INFO 2018/08/19 12:58:18 Started async operation: Create Volume >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 104.471µs >[heketi] INFO 2018/08/19 12:58:18 Creating brick 594612b877c5b8eb926c7f3ebfac20c5 >[heketi] INFO 2018/08/19 12:58:18 Creating brick b6352433cabed2d46eb18549e9e21c7b >[heketi] INFO 2018/08/19 12:58:18 Creating brick e4cfc6352537908f646bffd0fda3624f >[kubeexec] DEBUG 2018/08/19 12:58:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_46beb54fc781fdfcd258b75a217adbe1 --virtualsize 1048576K --name brick_46beb54fc781fdfcd258b75a217adbe1 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_46beb54fc781fdfcd258b75a217adbe1" created. >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 147.666µs >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_58ca7062c8bd0c92fb4fa224d2beab6c --virtualsize 1048576K --name brick_58ca7062c8bd0c92fb4fa224d2beab6c >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_58ca7062c8bd0c92fb4fa224d2beab6c" created. >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_7657b82acd9c3ae96bb43c8431698ea8 --virtualsize 1048576K --name brick_c5b42219b2fa5bd950b7368e915d11bb >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_c5b42219b2fa5bd950b7368e915d11bb" created. >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 127.249µs >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:19 Allocating brick set #0 >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_46beb54fc781fdfcd258b75a217adbe1 >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_46beb54fc781fdfcd258b75a217adbe1 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 152.152µs >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_46beb54fc781fdfcd258b75a217adbe1 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Completed 202 Accepted in 111.117197ms >[asynchttp] INFO 2018/08/19 12:58:19 asynchttp.go:288: Started job b59f14711b0bde6d723f2f4dd6b66d41 >[heketi] INFO 2018/08/19 12:58:19 Started async operation: Create Volume >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 127.755µs >[heketi] INFO 2018/08/19 12:58:19 Creating brick 0a5f1eb68d1989c9f203eaec05b7994b >[heketi] INFO 2018/08/19 12:58:19 Creating brick 001b531ee1519690498c59a97f6532fb >[heketi] INFO 2018/08/19 12:58:19 Creating brick c92163bc380b62931698e12d0d1bb6c3 >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_58ca7062c8bd0c92fb4fa224d2beab6c >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_58ca7062c8bd0c92fb4fa224d2beab6c isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c5b42219b2fa5bd950b7368e915d11bb >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c5b42219b2fa5bd950b7368e915d11bb isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 134.953µs >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_46beb54fc781fdfcd258b75a217adbe1 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_58ca7062c8bd0c92fb4fa224d2beab6c /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c5b42219b2fa5bd950b7368e915d11bb /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1/brick >Result: >[negroni] Started POST /volumes >[heketi] INFO 2018/08/19 12:58:19 Allocating brick set #0 >[negroni] Completed 202 Accepted in 22.82318ms >[asynchttp] INFO 2018/08/19 12:58:19 asynchttp.go:288: Started job cea00ffa6913cde01a94262c860f1502 >[heketi] INFO 2018/08/19 12:58:19 Started async operation: Create Volume >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 170.9µs >[heketi] INFO 2018/08/19 12:58:19 Creating brick 2260ab82f6401a9c4b3f38afe9b29525 >[heketi] INFO 2018/08/19 12:58:19 Creating brick d8d6c9b19f5ad36df62227e0685bb0c1 >[heketi] INFO 2018/08/19 12:58:19 Creating brick d1ad8b34a6d55dd94bd39af76f50798f >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c5b42219b2fa5bd950b7368e915d11bb /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb >Result: >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2002 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_58ca7062c8bd0c92fb4fa224d2beab6c /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 142.409µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 166.512µs >[kubeexec] DEBUG 2018/08/19 12:58:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb/brick >Result: >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 127.371µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1/brick >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 199.89µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2002 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 144.775µs >[negroni] Started POST /volumes >[heketi] WARNING 2018/08/19 12:58:20 operations in-flight (8) exceeds limit (8) >[negroni] Completed 429 Too Many Requests in 256.101µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb/brick >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 141.649µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 173.49µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_26a67e0af8755d7a1b5d94b44addf1ab --virtualsize 1048576K --name brick_26a67e0af8755d7a1b5d94b44addf1ab >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_26a67e0af8755d7a1b5d94b44addf1ab" created. >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 221.857µs >[negroni] Started POST /volumes >[heketi] WARNING 2018/08/19 12:58:20 operations in-flight (8) exceeds limit (8) >[negroni] Completed 429 Too Many Requests in 308.654µs >[kubeexec] DEBUG 2018/08/19 12:58:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_c913e8d56b361b0a337de2301ceaf0d1 --virtualsize 1048576K --name brick_f5124a847a7f09bce7401fdfd395280f >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_f5124a847a7f09bce7401fdfd395280f" created. >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 103.164µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 126.609µs >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 120.607µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_26a67e0af8755d7a1b5d94b44addf1ab >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_26a67e0af8755d7a1b5d94b44addf1ab isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_26a67e0af8755d7a1b5d94b44addf1ab /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 181.239µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_26a67e0af8755d7a1b5d94b44addf1ab /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_f5124a847a7f09bce7401fdfd395280f >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_f5124a847a7f09bce7401fdfd395280f isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 151.683µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_f5124a847a7f09bce7401fdfd395280f /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab/brick >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 160.271µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2003 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_f5124a847a7f09bce7401fdfd395280f /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 143.206µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2003 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 119.691µs >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 121.599µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 166.56µs >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 129.249µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee >Result: >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_106bd6171c93e30bd17842688b5cc99e --virtualsize 1048576K --name brick_106bd6171c93e30bd17842688b5cc99e >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_106bd6171c93e30bd17842688b5cc99e" created. >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 142.823µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_5a268d74742c57864d5dc7db3a16c2ee --virtualsize 1048576K --name brick_5a268d74742c57864d5dc7db3a16c2ee >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_5a268d74742c57864d5dc7db3a16c2ee" created. >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 145.616µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 116.146µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_106bd6171c93e30bd17842688b5cc99e >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_106bd6171c93e30bd17842688b5cc99e isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 138.106µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5a268d74742c57864d5dc7db3a16c2ee >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5a268d74742c57864d5dc7db3a16c2ee isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_106bd6171c93e30bd17842688b5cc99e /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5a268d74742c57864d5dc7db3a16c2ee /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 144.15µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_106bd6171c93e30bd17842688b5cc99e /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 140.528µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_5a268d74742c57864d5dc7db3a16c2ee /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee >Result: >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e/brick >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 163.973µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee/brick >Result: >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 103.61µs >[kubeexec] DEBUG 2018/08/19 12:58:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2004 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2004 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee/brick >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 116.113µs >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154 >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 179.431µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 130.186µs >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_901cfb8b10dd7a04d9fed63e1e5ab20b --virtualsize 1048576K --name brick_fb7cb684a9a164563eae1fb333da5088 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_fb7cb684a9a164563eae1fb333da5088" created. >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 132.779µs >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_67a0f677207005db8d0e662dcc651154 --virtualsize 1048576K --name brick_67a0f677207005db8d0e662dcc651154 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_67a0f677207005db8d0e662dcc651154" created. >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 126.66µs >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_67a0f677207005db8d0e662dcc651154 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_67a0f677207005db8d0e662dcc651154 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 122.369µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 130.545µs >[kubeexec] DEBUG 2018/08/19 12:58:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_67a0f677207005db8d0e662dcc651154 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 90.645µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_fb7cb684a9a164563eae1fb333da5088 >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_fb7cb684a9a164563eae1fb333da5088 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_67a0f677207005db8d0e662dcc651154 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_fb7cb684a9a164563eae1fb333da5088 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 160.117µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_fb7cb684a9a164563eae1fb333da5088 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2005 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088/brick >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 164.867µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154/brick >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 142.306µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2005 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f >Result: >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088/brick >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 123.669µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 150.722µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_4afe6d4a9a99294864dbd4874a0d6513 --virtualsize 1048576K --name brick_e4cfc6352537908f646bffd0fda3624f >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_e4cfc6352537908f646bffd0fda3624f" created. >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_6220bc2df0dad69f3f16527d535ebb64 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_7d13e6899ea05f456177c644bfd67224/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_6c1ce487d6ccf842287e2cabebc1c8f3/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5363271fa12ac2979f057f463927a91a/brick >Result: volume create: vol_6220bc2df0dad69f3f16527d535ebb64: success: please start the volume to access data >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 178.571µs >[kubeexec] DEBUG 2018/08/19 12:58:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c/brick >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 129.895µs >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 167.455µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2002 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_e4cfc6352537908f646bffd0fda3624f >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_e4cfc6352537908f646bffd0fda3624f isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:25 Creating volume vol_7d7297fc249953e9d3add7e45fc639ef replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_e4cfc6352537908f646bffd0fda3624f /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 100.675µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_e4cfc6352537908f646bffd0fda3624f /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 161.057µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f/brick >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 148.048µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2006 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_77c490cd008d35f5a4ae9fc53242e1a5 --virtualsize 1048576K --name brick_77c490cd008d35f5a4ae9fc53242e1a5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_77c490cd008d35f5a4ae9fc53242e1a5" created. >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 129.361µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3 >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 180.493µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 150.797µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 132.971µs >[kubeexec] DEBUG 2018/08/19 12:58:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_c92163bc380b62931698e12d0d1bb6c3 --virtualsize 1048576K --name brick_c92163bc380b62931698e12d0d1bb6c3 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_c92163bc380b62931698e12d0d1bb6c3" created. >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 139.838µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_77c490cd008d35f5a4ae9fc53242e1a5 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_77c490cd008d35f5a4ae9fc53242e1a5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_77c490cd008d35f5a4ae9fc53242e1a5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 121.544µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_77c490cd008d35f5a4ae9fc53242e1a5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c92163bc380b62931698e12d0d1bb6c3 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c92163bc380b62931698e12d0d1bb6c3 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 160.03µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c92163bc380b62931698e12d0d1bb6c3 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 116.742µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2003 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_c92163bc380b62931698e12d0d1bb6c3 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3 >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 153.215µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:26 Creating volume vol_89c28865e9b6d08752c87d47a90fdbd7 replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 99.45µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2007 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3/brick >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 122.274µs >[kubeexec] DEBUG 2018/08/19 12:58:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525 >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 96.534µs >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 152.802µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_8ce0ea78fb1da2a65ccf177e23145da2 --virtualsize 1048576K --name brick_49f8272024aac926535db90baff49a6e >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_49f8272024aac926535db90baff49a6e" created. >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 124.797µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_2260ab82f6401a9c4b3f38afe9b29525 --virtualsize 1048576K --name brick_2260ab82f6401a9c4b3f38afe9b29525 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_2260ab82f6401a9c4b3f38afe9b29525" created. >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_49f8272024aac926535db90baff49a6e >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_49f8272024aac926535db90baff49a6e isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 120.163µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_49f8272024aac926535db90baff49a6e /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 101.517µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_2260ab82f6401a9c4b3f38afe9b29525 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_2260ab82f6401a9c4b3f38afe9b29525 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_2260ab82f6401a9c4b3f38afe9b29525 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_49f8272024aac926535db90baff49a6e /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 92.861µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 130.308µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_2260ab82f6401a9c4b3f38afe9b29525 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2004 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:27 Creating volume vol_4593fe32d39663615b5c5d106c7ef15b replica 3 >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 99.691µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2008 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525/brick >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 136.668µs >[kubeexec] DEBUG 2018/08/19 12:58:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5 >Result: >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 200 OK in 88.964µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_6220bc2df0dad69f3f16527d535ebb64 >Result: volume start: vol_6220bc2df0dad69f3f16527d535ebb64: success >[heketi] INFO 2018/08/19 12:58:28 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:28 asynchttp.go:292: Completed job a60cecc2ac0a55a3d6a45bedd61db9c0 in 11.145008942s >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 163.847µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b >Result: >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_113d866e76ba6fecdb460f41734f57d5 --virtualsize 1048576K --name brick_113d866e76ba6fecdb460f41734f57d5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_113d866e76ba6fecdb460f41734f57d5" created. >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume create vol_7d7297fc249953e9d3add7e45fc639ef replica 3 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c5b42219b2fa5bd950b7368e915d11bb/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_46beb54fc781fdfcd258b75a217adbe1/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_58ca7062c8bd0c92fb4fa224d2beab6c/brick >Result: volume create: vol_7d7297fc249953e9d3add7e45fc639ef: success: please start the volume to access data >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 140.821µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 132.949µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_b6352433cabed2d46eb18549e9e21c7b --virtualsize 1048576K --name brick_b6352433cabed2d46eb18549e9e21c7b >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_b6352433cabed2d46eb18549e9e21c7b" created. >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 128.471µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_113d866e76ba6fecdb460f41734f57d5 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_113d866e76ba6fecdb460f41734f57d5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 105.008µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_113d866e76ba6fecdb460f41734f57d5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_b6352433cabed2d46eb18549e9e21c7b >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_b6352433cabed2d46eb18549e9e21c7b isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_b6352433cabed2d46eb18549e9e21c7b /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 114.672µs >[kubeexec] DEBUG 2018/08/19 12:58:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_113d866e76ba6fecdb460f41734f57d5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5 >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 142.974µs >[negroni] Started GET /queue/a60cecc2ac0a55a3d6a45bedd61db9c0 >[negroni] Completed 303 See Other in 122.462µs >[negroni] Started GET /volumes/6220bc2df0dad69f3f16527d535ebb64 >[negroni] Completed 200 OK in 4.516982ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 2.308202ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 3.449406ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 1.779323ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.412021ms >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_b6352433cabed2d46eb18549e9e21c7b /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b >Result: >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2005 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5/brick >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 119.351µs >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2006 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:29 Creating volume vol_ab454218439342563011d5adc5567ef2 replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5 >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 142.082µs >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 143.668µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 153.375µs >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_594612b877c5b8eb926c7f3ebfac20c5 --virtualsize 1048576K --name brick_594612b877c5b8eb926c7f3ebfac20c5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_594612b877c5b8eb926c7f3ebfac20c5" created. >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_a40c6f645c543b82bce7071e200acd2d --virtualsize 1048576K --name brick_0a5f1eb68d1989c9f203eaec05b7994b >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_0a5f1eb68d1989c9f203eaec05b7994b" created. >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 125.63µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 131.574µs >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_594612b877c5b8eb926c7f3ebfac20c5 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_594612b877c5b8eb926c7f3ebfac20c5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 144.524µs >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0a5f1eb68d1989c9f203eaec05b7994b >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0a5f1eb68d1989c9f203eaec05b7994b isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_594612b877c5b8eb926c7f3ebfac20c5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0a5f1eb68d1989c9f203eaec05b7994b /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_594612b877c5b8eb926c7f3ebfac20c5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5 >Result: >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 128.739µs >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0a5f1eb68d1989c9f203eaec05b7994b /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2006 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5/brick >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 148.734µs >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2007 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:30 Creating volume vol_620db35c5edc8a872685bd2a5d708f80 replica 3 >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 106.419µs >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb >Result: >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 141.057µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 135.448µs >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_001b531ee1519690498c59a97f6532fb --virtualsize 1048576K --name brick_001b531ee1519690498c59a97f6532fb >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_001b531ee1519690498c59a97f6532fb" created. >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 146.34µs >[kubeexec] DEBUG 2018/08/19 12:58:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_5653e0cfe3794e2487fb088739467be8 --virtualsize 1048576K --name brick_d1ad8b34a6d55dd94bd39af76f50798f >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_d1ad8b34a6d55dd94bd39af76f50798f" created. >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 122.385µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_001b531ee1519690498c59a97f6532fb >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_001b531ee1519690498c59a97f6532fb isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 107.747µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_d1ad8b34a6d55dd94bd39af76f50798f >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_d1ad8b34a6d55dd94bd39af76f50798f isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_001b531ee1519690498c59a97f6532fb /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_d1ad8b34a6d55dd94bd39af76f50798f /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 143.987µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_001b531ee1519690498c59a97f6532fb /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_d1ad8b34a6d55dd94bd39af76f50798f /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 105.994µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb/brick >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 119.513µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2008 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2007 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:31 Creating volume vol_73a91a908d197f91b7daefefa0d78d71 replica 3 >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 118.643µs >[kubeexec] DEBUG 2018/08/19 12:58:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1 >Result: >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 130.245µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 126.978µs >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_d8d6c9b19f5ad36df62227e0685bb0c1 --virtualsize 1048576K --name brick_d8d6c9b19f5ad36df62227e0685bb0c1 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_d8d6c9b19f5ad36df62227e0685bb0c1" created. >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 122.982µs >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d8d6c9b19f5ad36df62227e0685bb0c1 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d8d6c9b19f5ad36df62227e0685bb0c1 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 200 OK in 108.045µs >[negroni] Started POST /volumes >[negroni] Started POST /volumes >[heketi] WARNING 2018/08/19 12:58:32 operations in-flight (8) exceeds limit (8) >[negroni] Completed 429 Too Many Requests in 179.027µs >[heketi] INFO 2018/08/19 12:58:32 Allocating brick set #0 >[negroni] Completed 202 Accepted in 15.705732ms >[asynchttp] INFO 2018/08/19 12:58:32 asynchttp.go:288: Started job 7615ebd8c55d0fde3c5320f4f298e00a >[heketi] INFO 2018/08/19 12:58:32 Started async operation: Create Volume >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 109.052µs >[heketi] INFO 2018/08/19 12:58:32 Creating brick 0cba529336faa43a716a1cd51835a72e >[heketi] INFO 2018/08/19 12:58:32 Creating brick d3ee18d1d7d9c1c8c039948ae48af935 >[heketi] INFO 2018/08/19 12:58:32 Creating brick 44fcd9685f8ddd77ba6fa5a2f41706bf >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d8d6c9b19f5ad36df62227e0685bb0c1 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 87.7µs >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d8d6c9b19f5ad36df62227e0685bb0c1 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1 >Result: >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 135.344µs >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume start vol_7d7297fc249953e9d3add7e45fc639ef >Result: volume start: vol_7d7297fc249953e9d3add7e45fc639ef: success >[heketi] INFO 2018/08/19 12:58:32 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:32 asynchttp.go:292: Completed job 0e788e9dc8c7f27b4ed862a8b7bdee79 in 15.278562875s >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_ab454218439342563011d5adc5567ef2 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_fb7cb684a9a164563eae1fb333da5088/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_113d866e76ba6fecdb460f41734f57d5/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_67a0f677207005db8d0e662dcc651154/brick >Result: volume create: vol_ab454218439342563011d5adc5567ef2: success: please start the volume to access data >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 173.519µs >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2008 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume create vol_89c28865e9b6d08752c87d47a90fdbd7 replica 3 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f5124a847a7f09bce7401fdfd395280f/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_26a67e0af8755d7a1b5d94b44addf1ab/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_77c490cd008d35f5a4ae9fc53242e1a5/brick >Result: volume create: vol_89c28865e9b6d08752c87d47a90fdbd7: success: please start the volume to access data >[kubeexec] DEBUG 2018/08/19 12:58:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:32 Creating volume vol_f4c046284f1be3b0a332a8cf5d1e2af8 replica 3 >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 101.826µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 121.298µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 117.049µs >[negroni] Started GET /queue/0e788e9dc8c7f27b4ed862a8b7bdee79 >[negroni] Completed 303 See Other in 169.084µs >[negroni] Started GET /volumes/7d7297fc249953e9d3add7e45fc639ef >[negroni] Completed 200 OK in 3.397618ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 2.14151ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 3.498124ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 2.024966ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.167792ms >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 163.714µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 106.226µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 129.156µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 141.024µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 106.785µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 168.416µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 123.933µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 119.642µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 125.331µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 183.564µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 124.309µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 141.451µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 120.936µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 120.511µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 167.393µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 141.747µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 212.584µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 120.175µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 129.359µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 112.404µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 138.823µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 190.467µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 142.94µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 132.925µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 116.75µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 202.775µs >[heketi] INFO 2018/08/19 12:58:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 12:58:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 123.484µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 161.051µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 137.219µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 85.926µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 179.039µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 119.22µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 110.407µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 141.135µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 113.331µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 175.613µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 124.637µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 200 OK in 133.424µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 188.554µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 134.361µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 120.874µs >[kubeexec] DEBUG 2018/08/19 12:58:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_ab454218439342563011d5adc5567ef2 >Result: volume start: vol_ab454218439342563011d5adc5567ef2: success >[heketi] INFO 2018/08/19 12:58:39 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:39 asynchttp.go:292: Completed job 06bc4d4c3e6cf2cc2339a64c9d216102 in 20.580800917s >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 128.082µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 137.127µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 86.377µs >[negroni] Started GET /queue/06bc4d4c3e6cf2cc2339a64c9d216102 >[negroni] Completed 303 See Other in 145.533µs >[negroni] Started GET /volumes/ab454218439342563011d5adc5567ef2 >[negroni] Completed 200 OK in 5.186523ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 2.004701ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 4.489043ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 1.979184ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.766326ms >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 131.413µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 200 OK in 185.968µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 246.674µs >[kubeexec] DEBUG 2018/08/19 12:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: gluster --mode=script volume create vol_4593fe32d39663615b5c5d106c7ef15b replica 3 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_49f8272024aac926535db90baff49a6e/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_5a268d74742c57864d5dc7db3a16c2ee/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_106bd6171c93e30bd17842688b5cc99e/brick >Result: volume create: vol_4593fe32d39663615b5c5d106c7ef15b: success: please start the volume to access data >[kubeexec] DEBUG 2018/08/19 12:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume start vol_89c28865e9b6d08752c87d47a90fdbd7 >Result: volume start: vol_89c28865e9b6d08752c87d47a90fdbd7: success >[heketi] INFO 2018/08/19 12:58:40 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:40 asynchttp.go:292: Completed job dcc0e27239094affd743114ff7124193 in 22.291747927s >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 144.588µs >[kubeexec] DEBUG 2018/08/19 12:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_73a91a908d197f91b7daefefa0d78d71 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0a5f1eb68d1989c9f203eaec05b7994b/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_001b531ee1519690498c59a97f6532fb/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_c92163bc380b62931698e12d0d1bb6c3/brick >Result: volume create: vol_73a91a908d197f91b7daefefa0d78d71: success: please start the volume to access data >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 133.293µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 222.477µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 183.439µs >[negroni] Started GET /queue/dcc0e27239094affd743114ff7124193 >[negroni] Completed 303 See Other in 182.972µs >[negroni] Started GET /volumes/89c28865e9b6d08752c87d47a90fdbd7 >[negroni] Completed 200 OK in 5.048584ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 1.527718ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 5.963383ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 1.893723ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.275752ms >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 123.21µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 144.844µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 119.719µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 89.853µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 177.008µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 137.316µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 116.179µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 126.022µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 179.539µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 127.601µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 141.405µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 135.41µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 149.376µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 69.875µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 188.158µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 217.058µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 123.001µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 116.117µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 126.343µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 164.038µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 140.269µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 200 OK in 145.811µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 121.781µs >[kubeexec] DEBUG 2018/08/19 12:58:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: gluster --mode=script volume start vol_4593fe32d39663615b5c5d106c7ef15b >Result: volume start: vol_4593fe32d39663615b5c5d106c7ef15b: success >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 149.684µs >[heketi] INFO 2018/08/19 12:58:45 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:45 asynchttp.go:292: Completed job 86879cb72a8a124f762947bebb132c93 in 27.334809877s >[kubeexec] DEBUG 2018/08/19 12:58:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935 >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 123.768µs >[kubeexec] DEBUG 2018/08/19 12:58:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_d3ee18d1d7d9c1c8c039948ae48af935 --virtualsize 1048576K --name brick_d3ee18d1d7d9c1c8c039948ae48af935 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_d3ee18d1d7d9c1c8c039948ae48af935" created. >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 115.428µs >[negroni] Started GET /queue/86879cb72a8a124f762947bebb132c93 >[negroni] Completed 303 See Other in 130.999µs >[negroni] Started GET /volumes/4593fe32d39663615b5c5d106c7ef15b >[negroni] Completed 200 OK in 4.639709ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 1.083381ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d3ee18d1d7d9c1c8c039948ae48af935 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d3ee18d1d7d9c1c8c039948ae48af935 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Completed 200 OK in 4.277101ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 2.306706ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.159911ms >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d3ee18d1d7d9c1c8c039948ae48af935 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 173.028µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 200 OK in 183.947µs >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_d3ee18d1d7d9c1c8c039948ae48af935 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume create vol_620db35c5edc8a872685bd2a5d708f80 replica 3 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_e4cfc6352537908f646bffd0fda3624f/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_b6352433cabed2d46eb18549e9e21c7b/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_594612b877c5b8eb926c7f3ebfac20c5/brick >Result: volume create: vol_620db35c5edc8a872685bd2a5d708f80: success: please start the volume to access data >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_73a91a908d197f91b7daefefa0d78d71 >Result: volume start: vol_73a91a908d197f91b7daefefa0d78d71: success >[heketi] INFO 2018/08/19 12:58:46 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:46 asynchttp.go:292: Completed job b59f14711b0bde6d723f2f4dd6b66d41 in 27.106701769s >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e >Result: >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2009 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 153.004µs >[kubeexec] DEBUG 2018/08/19 12:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_c700497c94fa9d3a557327d491c9bfda --virtualsize 1048576K --name brick_0cba529336faa43a716a1cd51835a72e >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_0cba529336faa43a716a1cd51835a72e" created. >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 117.858µs >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0cba529336faa43a716a1cd51835a72e >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0cba529336faa43a716a1cd51835a72e isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[negroni] Started POST /volumes >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0cba529336faa43a716a1cd51835a72e /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[heketi] INFO 2018/08/19 12:58:47 Allocating brick set #0 >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 157.754µs >[negroni] Started GET /queue/b59f14711b0bde6d723f2f4dd6b66d41 >[negroni] Completed 303 See Other in 104.064µs >[negroni] Started GET /volumes/73a91a908d197f91b7daefefa0d78d71 >[negroni] Completed 200 OK in 2.368481ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 1.249804ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 4.001433ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 2.543858ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.192293ms >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_0cba529336faa43a716a1cd51835a72e /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e >Result: >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 135.974µs >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2009 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e/brick >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 138.739µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 148.942µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 142.293µs >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 170.552µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 149.422µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 184.109µs >[kubeexec] DEBUG 2018/08/19 12:58:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume start vol_620db35c5edc8a872685bd2a5d708f80 >Result: volume start: vol_620db35c5edc8a872685bd2a5d708f80: success >[kubeexec] DEBUG 2018/08/19 12:58:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_f4c046284f1be3b0a332a8cf5d1e2af8 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d1ad8b34a6d55dd94bd39af76f50798f/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d8d6c9b19f5ad36df62227e0685bb0c1/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_2260ab82f6401a9c4b3f38afe9b29525/brick >Result: volume create: vol_f4c046284f1be3b0a332a8cf5d1e2af8: success: please start the volume to access data >[kubeexec] DEBUG 2018/08/19 12:58:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 148.45µs >[kubeexec] DEBUG 2018/08/19 12:58:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_44fcd9685f8ddd77ba6fa5a2f41706bf --virtualsize 1048576K --name brick_44fcd9685f8ddd77ba6fa5a2f41706bf >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_44fcd9685f8ddd77ba6fa5a2f41706bf" created. >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 247.2µs >[kubeexec] DEBUG 2018/08/19 12:58:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_44fcd9685f8ddd77ba6fa5a2f41706bf >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_44fcd9685f8ddd77ba6fa5a2f41706bf isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_44fcd9685f8ddd77ba6fa5a2f41706bf /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 125.886µs >[kubeexec] DEBUG 2018/08/19 12:58:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_44fcd9685f8ddd77ba6fa5a2f41706bf /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf >Result: >[kubeexec] DEBUG 2018/08/19 12:58:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2009 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf/brick >Result: >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 200 OK in 117.194µs >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:51 Creating volume vol_22a62b829764dab3004e2e9ccaa21770 replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11177 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 12:58:51 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 12:58:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Completed 202 Accepted in 3.836092017s >[asynchttp] INFO 2018/08/19 12:58:51 asynchttp.go:288: Started job 95d13197fe79b3eed127630fb283b862 >[heketi] INFO 2018/08/19 12:58:51 Started async operation: Create Volume >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 114.868µs >[heketi] INFO 2018/08/19 12:58:51 Creating brick 94a628d95e20b2650c4dbce3b4adb8a2 >[heketi] INFO 2018/08/19 12:58:51 Creating brick 03bfa3e8ed2a6191cc6727700878afff >[heketi] INFO 2018/08/19 12:58:51 Creating brick 359a1fb6e102c9257e365e5bd4bcdaf5 >[heketi] INFO 2018/08/19 12:58:51 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:51 asynchttp.go:292: Completed job e1294465722eb6f8c1c2842bde328ecf in 32.415925253s >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ9756 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 12:58:51 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 12:58:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir -p /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2 >Result: >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 184.134µs >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir -p /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_f89b9b3b7340e500f2c6367273182b28/tp_d12676a407950ee50dd084ad8a35986a --virtualsize 1048576K --name brick_94a628d95e20b2650c4dbce3b4adb8a2 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_94a628d95e20b2650c4dbce3b4adb8a2" created. >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 139.92µs >[kubeexec] DEBUG 2018/08/19 12:58:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_af2c81ded89e5ffa84b759f5fc717bc7/tp_359a1fb6e102c9257e365e5bd4bcdaf5 --virtualsize 1048576K --name brick_359a1fb6e102c9257e365e5bd4bcdaf5 >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_359a1fb6e102c9257e365e5bd4bcdaf5" created. >[negroni] Started GET /queue/e1294465722eb6f8c1c2842bde328ecf >[negroni] Completed 303 See Other in 134.408µs >[negroni] Started GET /volumes/620db35c5edc8a872685bd2a5d708f80 >[negroni] Completed 200 OK in 4.548612ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 1.457703ms >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 4.553247ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 2.660698ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.469652ms >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_94a628d95e20b2650c4dbce3b4adb8a2 >Result: meta-data=/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_94a628d95e20b2650c4dbce3b4adb8a2 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: awk "BEGIN {print \"/dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_94a628d95e20b2650c4dbce3b4adb8a2 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 150.23µs >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_359a1fb6e102c9257e365e5bd4bcdaf5 >Result: meta-data=/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_359a1fb6e102c9257e365e5bd4bcdaf5 isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: awk "BEGIN {print \"/dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_359a1fb6e102c9257e365e5bd4bcdaf5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5 xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_f89b9b3b7340e500f2c6367273182b28-brick_94a628d95e20b2650c4dbce3b4adb8a2 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2 >Result: >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 145.551µs >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: mkdir /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_359a1fb6e102c9257e365e5bd4bcdaf5 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5 >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chown :2010 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: mkdir /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: chmod 2775 /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chown :2010 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5/brick >Result: >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 131.46µs >[kubeexec] DEBUG 2018/08/19 12:58:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: chmod 2775 /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5/brick >Result: >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 124.385µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 139.278µs >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 200 OK in 120.221µs >[kubeexec] DEBUG 2018/08/19 12:58:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_f4c046284f1be3b0a332a8cf5d1e2af8 >Result: volume start: vol_f4c046284f1be3b0a332a8cf5d1e2af8: success >[heketi] INFO 2018/08/19 12:58:54 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:54 asynchttp.go:292: Completed job cea00ffa6913cde01a94262c860f1502 in 34.409816426s >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 120.28µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 130.786µs >[kubeexec] DEBUG 2018/08/19 12:58:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume create vol_22a62b829764dab3004e2e9ccaa21770 replica 3 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_0cba529336faa43a716a1cd51835a72e/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_d3ee18d1d7d9c1c8c039948ae48af935/brick 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_44fcd9685f8ddd77ba6fa5a2f41706bf/brick >Result: volume create: vol_22a62b829764dab3004e2e9ccaa21770: success: please start the volume to access data >[negroni] Started GET /queue/cea00ffa6913cde01a94262c860f1502 >[negroni] Completed 303 See Other in 134.11µs >[negroni] Started GET /volumes/f4c046284f1be3b0a332a8cf5d1e2af8 >[negroni] Completed 200 OK in 5.453934ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 945.608µs >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 4.866714ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 6.254879ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.707539ms >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 139.139µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 196.323µs >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 149.318µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 185.429µs >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 123.205µs >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 200 OK in 227.493µs >[kubeexec] DEBUG 2018/08/19 12:58:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: gluster --mode=script volume start vol_22a62b829764dab3004e2e9ccaa21770 >Result: volume start: vol_22a62b829764dab3004e2e9ccaa21770: success >[heketi] INFO 2018/08/19 12:58:57 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:58:57 asynchttp.go:292: Completed job 7615ebd8c55d0fde3c5320f4f298e00a in 25.396461321s >[kubeexec] DEBUG 2018/08/19 12:58:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir -p /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff >Result: >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 133.841µs >[kubeexec] DEBUG 2018/08/19 12:58:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: lvcreate --autobackup=n --poolmetadatasize 8192K --chunksize 256K --size 1048576K --thin vg_3fecdb9fc84903807eef185ca1057930/tp_03bfa3e8ed2a6191cc6727700878afff --virtualsize 1048576K --name brick_03bfa3e8ed2a6191cc6727700878afff >Result: Thin pool volume with chunk size 256.00 KiB can address at most 63.25 TiB of data. > Logical volume "brick_03bfa3e8ed2a6191cc6727700878afff" created. >[negroni] Started GET /queue/7615ebd8c55d0fde3c5320f4f298e00a >[negroni] Completed 303 See Other in 196.126µs >[negroni] Started GET /volumes/22a62b829764dab3004e2e9ccaa21770 >[negroni] Completed 200 OK in 5.201134ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 367.496µs >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 5.478147ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 3.228619ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.515138ms >[kubeexec] DEBUG 2018/08/19 12:58:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkfs.xfs -i size=512 -n size=8192 /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_03bfa3e8ed2a6191cc6727700878afff >Result: meta-data=/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_03bfa3e8ed2a6191cc6727700878afff isize=512 agcount=8, agsize=32768 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=0, sparse=0 >data = bsize=4096 blocks=262144, imaxpct=25 > = sunit=64 swidth=64 blks >naming =version 2 bsize=8192 ascii-ci=0 ftype=1 >log =internal log bsize=4096 blocks=2560, version=2 > = sectsz=512 sunit=64 blks, lazy-count=1 >realtime =none extsz=4096 blocks=0, rtextents=0 >[kubeexec] DEBUG 2018/08/19 12:58:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: awk "BEGIN {print \"/dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_03bfa3e8ed2a6191cc6727700878afff /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff xfs rw,inode64,noatime,nouuid 1 2\" >> \"/var/lib/heketi/fstab\"}" >Result: >[kubeexec] DEBUG 2018/08/19 12:58:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mount -o rw,inode64,noatime,nouuid /dev/mapper/vg_3fecdb9fc84903807eef185ca1057930-brick_03bfa3e8ed2a6191cc6727700878afff /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff >Result: >[kubeexec] DEBUG 2018/08/19 12:58:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: mkdir /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff/brick >Result: >[kubeexec] DEBUG 2018/08/19 12:58:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chown :2010 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff/brick >Result: >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 187.007µs >[kubeexec] DEBUG 2018/08/19 12:58:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: chmod 2775 /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff/brick >Result: >[cmdexec] INFO 2018/08/19 12:58:59 Creating volume vol_36bafa7f578d5d53ed221357b23571fe replica 3 >[kubeexec] DEBUG 2018/08/19 12:58:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 2min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ10085 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 12:58:59 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 12:58:59 Cleaned 0 nodes from health cache >[kubeexec] DEBUG 2018/08/19 12:58:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume create vol_36bafa7f578d5d53ed221357b23571fe replica 3 10.70.47.183:/var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_94a628d95e20b2650c4dbce3b4adb8a2/brick 10.70.47.54:/var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_03bfa3e8ed2a6191cc6727700878afff/brick 10.70.46.152:/var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_359a1fb6e102c9257e365e5bd4bcdaf5/brick >Result: volume create: vol_36bafa7f578d5d53ed221357b23571fe: success: please start the volume to access data >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 166.428µs >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 200 OK in 142.405µs >[kubeexec] DEBUG 2018/08/19 12:59:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster --mode=script volume start vol_36bafa7f578d5d53ed221357b23571fe >Result: volume start: vol_36bafa7f578d5d53ed221357b23571fe: success >[heketi] INFO 2018/08/19 12:59:01 Create Volume succeeded >[asynchttp] INFO 2018/08/19 12:59:01 asynchttp.go:292: Completed job 95d13197fe79b3eed127630fb283b862 in 10.692079784s >[negroni] Started GET /queue/95d13197fe79b3eed127630fb283b862 >[negroni] Completed 303 See Other in 175.661µs >[negroni] Started GET /volumes/36bafa7f578d5d53ed221357b23571fe >[negroni] Completed 200 OK in 3.909885ms >[negroni] Started GET /clusters/1e42dd6f356d6cf93b376a9ec0c824e7 >[negroni] Completed 200 OK in 299.412µs >[negroni] Started GET /nodes/5f0be17866654828d51d78022320f1f8 >[negroni] Completed 200 OK in 3.708344ms >[negroni] Started GET /nodes/6120e5eb7b35b19b1389d33ad1cf9991 >[negroni] Completed 200 OK in 2.741286ms >[negroni] Started GET /nodes/9889f77e242b9e5b2edc1f07ea800b87 >[negroni] Completed 200 OK in 1.30024ms >[heketi] INFO 2018/08/19 13:00:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:00:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:00:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:00:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:00:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:00:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:00:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-strmr Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 4min ago > Process: 218 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 220 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda557649a_a176_11e8_9e7f_005056a51f55.slice/docker-7deb2c18e6521a4619b6425c8697362063048d3aa259ba3d5fec00ad05ebbc2a.scope/system.slice/glusterd.service > ââ 220 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 455 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 463 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ10271 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d >[heketi] INFO 2018/08/19 13:00:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:00:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:02:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:02:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:02:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:02:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:02:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:02:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 3s ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:02:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:02:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:04:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:04:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:04:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:04:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:04:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:04:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 2min 3s ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:04:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:04:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:06:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:06:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:06:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:06:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:06:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:06:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 4min 3s ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:06:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:06:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:08:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:08:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:08:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:08:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:08:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:08:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:08:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:08:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.389454ms >[asynchttp] INFO 2018/08/19 13:09:45 asynchttp.go:288: Started job 109b98df95e44f659f253a00f98896c0 >[heketi] INFO 2018/08/19 13:09:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/109b98df95e44f659f253a00f98896c0 >[negroni] Completed 200 OK in 124.163µs >[cmdexec] INFO 2018/08/19 13:09:45 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:45 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 15.910779ms >[asynchttp] INFO 2018/08/19 13:09:46 asynchttp.go:288: Started job 89e63321855ea37a76124ca0264d504a >[heketi] INFO 2018/08/19 13:09:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/89e63321855ea37a76124ca0264d504a >[negroni] Completed 200 OK in 132.694µs >[cmdexec] INFO 2018/08/19 13:09:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/109b98df95e44f659f253a00f98896c0 >[negroni] Completed 200 OK in 185.492µs >[kubeexec] DEBUG 2018/08/19 13:09:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_adf94878662fea2994559d8a2bcd9ae3 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:478499b9-7c90-46b3-b5fb-fcef652b13a3", "USERNAME": "478499b9-7c90-46b3-b5fb-fcef652b13a3", "PASSWORD": "1ac6aa9a-9d38-4f3e-9501-6bafae44bece", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/89e63321855ea37a76124ca0264d504a >[negroni] Completed 200 OK in 134.033µs >[heketi] INFO 2018/08/19 13:09:47 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:47 asynchttp.go:292: Completed job 109b98df95e44f659f253a00f98896c0 in 1.531260788s >[kubeexec] DEBUG 2018/08/19 13:09:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/109b98df95e44f659f253a00f98896c0 >[negroni] Completed 303 See Other in 138.93µs >[negroni] Started GET /blockvolumes/adf94878662fea2994559d8a2bcd9ae3 >[negroni] Completed 200 OK in 2.053593ms >[negroni] Started GET /queue/89e63321855ea37a76124ca0264d504a >[negroni] Completed 200 OK in 221.465µs >[kubeexec] DEBUG 2018/08/19 13:09:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_45d9d7d74c88d683b6a82c9ed0d2c4f5 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7a608242-380f-4986-8496-84e1dfd52abe", "USERNAME": "7a608242-380f-4986-8496-84e1dfd52abe", "PASSWORD": "d2a66725-a402-4d04-988f-0fbc5e8a0d6e", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:09:48 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:48 asynchttp.go:292: Completed job 89e63321855ea37a76124ca0264d504a in 2.215408637s >[negroni] Started GET /queue/89e63321855ea37a76124ca0264d504a >[negroni] Completed 303 See Other in 164.143µs >[negroni] Started GET /blockvolumes/45d9d7d74c88d683b6a82c9ed0d2c4f5 >[negroni] Completed 200 OK in 2.145904ms >[negroni] Started POST /blockvolumes >[asynchttp] INFO 2018/08/19 13:09:49 asynchttp.go:288: Started job 348ed9f5e0862e272bf7e11d9982c729 >[heketi] INFO 2018/08/19 13:09:49 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 77.655425ms >[negroni] Started GET /queue/348ed9f5e0862e272bf7e11d9982c729 >[negroni] Completed 200 OK in 151.18µs >[cmdexec] INFO 2018/08/19 13:09:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:50 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/348ed9f5e0862e272bf7e11d9982c729 >[negroni] Completed 200 OK in 166.42µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 23.840178ms >[asynchttp] INFO 2018/08/19 13:09:51 asynchttp.go:288: Started job 49b782fa1afd62f02083a7c10606743e >[heketi] INFO 2018/08/19 13:09:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/49b782fa1afd62f02083a7c10606743e >[negroni] Completed 200 OK in 106.419µs >[cmdexec] INFO 2018/08/19 13:09:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_eb7327d499dd4d17d39a3b1a93fae091 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:25bc59c8-60e9-427c-99a5-fc3d6cd9df10", "USERNAME": "25bc59c8-60e9-427c-99a5-fc3d6cd9df10", "PASSWORD": "bce27ada-b593-4716-b6c1-126cba6baaea", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:09:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:51 asynchttp.go:292: Completed job 348ed9f5e0862e272bf7e11d9982c729 in 1.776364468s >[negroni] Started GET /queue/348ed9f5e0862e272bf7e11d9982c729 >[negroni] Completed 303 See Other in 134.175µs >[negroni] Started GET /blockvolumes/eb7327d499dd4d17d39a3b1a93fae091 >[negroni] Completed 200 OK in 2.361315ms >[kubeexec] DEBUG 2018/08/19 13:09:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/49b782fa1afd62f02083a7c10606743e >[negroni] Completed 200 OK in 121.219µs >[kubeexec] DEBUG 2018/08/19 13:09:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_5ad94efe0a937962cb090c3191888b8a ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:cfd51d96-95c6-410b-ae08-f413a5e452dd", "USERNAME": "cfd51d96-95c6-410b-ae08-f413a5e452dd", "PASSWORD": "d4e68e5e-094c-407c-8018-dd8b99a2a6c2", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:09:53 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:53 asynchttp.go:292: Completed job 49b782fa1afd62f02083a7c10606743e in 1.730472857s >[negroni] Started GET /queue/49b782fa1afd62f02083a7c10606743e >[negroni] Completed 303 See Other in 163.48µs >[negroni] Started GET /blockvolumes/5ad94efe0a937962cb090c3191888b8a >[negroni] Completed 200 OK in 2.130597ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 180.311471ms >[asynchttp] INFO 2018/08/19 13:09:55 asynchttp.go:288: Started job 1957ab88c8d27b980ceb5e78e0b8e073 >[heketi] INFO 2018/08/19 13:09:55 Started async operation: Create Block Volume >[negroni] Started GET /queue/1957ab88c8d27b980ceb5e78e0b8e073 >[negroni] Completed 200 OK in 212.445µs >[cmdexec] INFO 2018/08/19 13:09:55 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1957ab88c8d27b980ceb5e78e0b8e073 >[negroni] Completed 200 OK in 206.245µs >[kubeexec] DEBUG 2018/08/19 13:09:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_a4d330f02a7e2a9abf83f761cb106932 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:23e4dbec-17f6-4b09-8f1e-371ddb9449d7", "USERNAME": "23e4dbec-17f6-4b09-8f1e-371ddb9449d7", "PASSWORD": "14118236-520a-4ffe-a832-5b8751eb6ff3", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:09:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:57 asynchttp.go:292: Completed job 1957ab88c8d27b980ceb5e78e0b8e073 in 1.570149693s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 20.185294ms >[asynchttp] INFO 2018/08/19 13:09:57 asynchttp.go:288: Started job de7503a480f56e0350cbfff666f730de >[heketi] INFO 2018/08/19 13:09:57 Started async operation: Create Block Volume >[negroni] Started GET /queue/de7503a480f56e0350cbfff666f730de >[negroni] Completed 200 OK in 121.408µs >[cmdexec] INFO 2018/08/19 13:09:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:09:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:09:57 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1957ab88c8d27b980ceb5e78e0b8e073 >[negroni] Completed 303 See Other in 127.895µs >[negroni] Started GET /blockvolumes/a4d330f02a7e2a9abf83f761cb106932 >[negroni] Completed 200 OK in 1.606798ms >[kubeexec] DEBUG 2018/08/19 13:09:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:09:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:09:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/de7503a480f56e0350cbfff666f730de >[negroni] Completed 200 OK in 172.424µs >[kubeexec] DEBUG 2018/08/19 13:09:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c48949212f25c26dabaf72b185962875 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:aca3cc0a-e1a9-488d-92fc-94b0fa082bae", "USERNAME": "aca3cc0a-e1a9-488d-92fc-94b0fa082bae", "PASSWORD": "dad78e06-c382-44ac-b2ef-7cb989da400f", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:09:58 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:09:58 asynchttp.go:292: Completed job de7503a480f56e0350cbfff666f730de in 1.639264516s >[negroni] Started GET /queue/de7503a480f56e0350cbfff666f730de >[negroni] Completed 303 See Other in 152.713µs >[negroni] Started GET /blockvolumes/c48949212f25c26dabaf72b185962875 >[negroni] Completed 200 OK in 2.574553ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 18.770536ms >[asynchttp] INFO 2018/08/19 13:10:01 asynchttp.go:288: Started job a988afdba848d2473302b09b978da6fb >[heketi] INFO 2018/08/19 13:10:01 Started async operation: Create Block Volume >[negroni] Started GET /queue/a988afdba848d2473302b09b978da6fb >[negroni] Completed 200 OK in 171.657µs >[cmdexec] INFO 2018/08/19 13:10:01 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a988afdba848d2473302b09b978da6fb >[negroni] Completed 200 OK in 121.336µs >[kubeexec] DEBUG 2018/08/19 13:10:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_f19a77babfe986de7164bcf2c1b43414 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b2d688a0-4490-42c2-bca4-1de2050f5506", "USERNAME": "b2d688a0-4490-42c2-bca4-1de2050f5506", "PASSWORD": "cfd8339b-3fc2-458a-8f4e-515bf7466f21", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:02 asynchttp.go:292: Completed job a988afdba848d2473302b09b978da6fb in 1.61635134s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 19.244757ms >[asynchttp] INFO 2018/08/19 13:10:02 asynchttp.go:288: Started job c381e1934946c2483a16f46c7652af7f >[heketi] INFO 2018/08/19 13:10:02 Started async operation: Create Block Volume >[negroni] Started GET /queue/c381e1934946c2483a16f46c7652af7f >[negroni] Completed 200 OK in 103.748µs >[cmdexec] INFO 2018/08/19 13:10:02 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a988afdba848d2473302b09b978da6fb >[negroni] Completed 303 See Other in 196.525µs >[negroni] Started GET /blockvolumes/f19a77babfe986de7164bcf2c1b43414 >[negroni] Completed 200 OK in 1.81531ms >[kubeexec] DEBUG 2018/08/19 13:10:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c381e1934946c2483a16f46c7652af7f >[negroni] Completed 200 OK in 140.59µs >[kubeexec] DEBUG 2018/08/19 13:10:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_522a631a8cccde15bdc5a5918c20e461 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:61995f1e-e401-4348-b83d-78430fcc53ea", "USERNAME": "61995f1e-e401-4348-b83d-78430fcc53ea", "PASSWORD": "84624f94-e748-4067-b409-eccb4cd4aea6", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:04 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:04 asynchttp.go:292: Completed job c381e1934946c2483a16f46c7652af7f in 1.704803284s >[negroni] Started GET /queue/c381e1934946c2483a16f46c7652af7f >[negroni] Completed 303 See Other in 123.71µs >[negroni] Started GET /blockvolumes/522a631a8cccde15bdc5a5918c20e461 >[negroni] Completed 200 OK in 3.404206ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 60.475334ms >[asynchttp] INFO 2018/08/19 13:10:06 asynchttp.go:288: Started job aead3783da53c8b25ad4e161ca2651ba >[heketi] INFO 2018/08/19 13:10:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/aead3783da53c8b25ad4e161ca2651ba >[negroni] Completed 200 OK in 140.883µs >[cmdexec] INFO 2018/08/19 13:10:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:07 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/aead3783da53c8b25ad4e161ca2651ba >[negroni] Completed 200 OK in 151.652µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 23.665298ms >[asynchttp] INFO 2018/08/19 13:10:08 asynchttp.go:288: Started job 7003dc1cf7bdc2fc8bc81f5080a1c142 >[heketi] INFO 2018/08/19 13:10:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/7003dc1cf7bdc2fc8bc81f5080a1c142 >[negroni] Completed 200 OK in 128.068µs >[cmdexec] INFO 2018/08/19 13:10:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_ee841961a6f65a10c94fdb0a5c7aec3d ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:85e8371b-41be-4b61-8198-423ad6e2e021", "USERNAME": "85e8371b-41be-4b61-8198-423ad6e2e021", "PASSWORD": "70f2d9f3-7740-4cd0-9198-5113cbcb2b40", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 13:10:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 13:10:08 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:08 asynchttp.go:292: Completed job aead3783da53c8b25ad4e161ca2651ba in 1.9597763s >[negroni] Started GET /queue/aead3783da53c8b25ad4e161ca2651ba >[negroni] Completed 303 See Other in 182.413µs >[negroni] Started GET /blockvolumes/ee841961a6f65a10c94fdb0a5c7aec3d >[negroni] Completed 200 OK in 2.860346ms >[kubeexec] DEBUG 2018/08/19 13:10:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7003dc1cf7bdc2fc8bc81f5080a1c142 >[negroni] Completed 200 OK in 134.232µs >[kubeexec] DEBUG 2018/08/19 13:10:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_4097417da82a9f005328db2ec4d5ca3b ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:26498cf7-3d78-4971-b69b-7db2479b90c3", "USERNAME": "26498cf7-3d78-4971-b69b-7db2479b90c3", "PASSWORD": "661d4ce7-beb6-474b-acc5-e2c08d337246", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:10 asynchttp.go:292: Completed job 7003dc1cf7bdc2fc8bc81f5080a1c142 in 1.763368198s >[negroni] Started GET /queue/7003dc1cf7bdc2fc8bc81f5080a1c142 >[negroni] Completed 303 See Other in 186.024µs >[negroni] Started GET /blockvolumes/4097417da82a9f005328db2ec4d5ca3b >[negroni] Completed 200 OK in 2.92539ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 19.740367ms >[asynchttp] INFO 2018/08/19 13:10:12 asynchttp.go:288: Started job 1da0f23231ec8cfc95354aff3f6eb367 >[heketi] INFO 2018/08/19 13:10:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/1da0f23231ec8cfc95354aff3f6eb367 >[negroni] Completed 200 OK in 111.906µs >[cmdexec] INFO 2018/08/19 13:10:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:12 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:12 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1da0f23231ec8cfc95354aff3f6eb367 >[negroni] Completed 200 OK in 128.099µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 22.447803ms >[asynchttp] INFO 2018/08/19 13:10:14 asynchttp.go:288: Started job a1ee9108850351dfccbc52b87ed7b59a >[heketi] INFO 2018/08/19 13:10:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/a1ee9108850351dfccbc52b87ed7b59a >[negroni] Completed 200 OK in 154.257µs >[cmdexec] INFO 2018/08/19 13:10:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_335be84b264b64edf01beebdb57422a6 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b6cefe6b-a3f9-4027-bdc6-32b692f770e9", "USERNAME": "b6cefe6b-a3f9-4027-bdc6-32b692f770e9", "PASSWORD": "c3d1391a-b03c-42ba-a41d-dc715fb36f09", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:14 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:14 asynchttp.go:292: Completed job 1da0f23231ec8cfc95354aff3f6eb367 in 1.864714468s >[kubeexec] DEBUG 2018/08/19 13:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:14 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1da0f23231ec8cfc95354aff3f6eb367 >[negroni] Completed 303 See Other in 168.866µs >[negroni] Started GET /blockvolumes/335be84b264b64edf01beebdb57422a6 >[negroni] Completed 200 OK in 2.079895ms >[kubeexec] DEBUG 2018/08/19 13:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a1ee9108850351dfccbc52b87ed7b59a >[negroni] Completed 200 OK in 139.449µs >[kubeexec] DEBUG 2018/08/19 13:10:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c09c849b291f015f8d35bdc68e9dca65 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4e1f4434-fe0e-4b3f-bf8b-12c5164b60dd", "USERNAME": "4e1f4434-fe0e-4b3f-bf8b-12c5164b60dd", "PASSWORD": "255ba0d9-cda9-4a10-bd20-b0e9505748e0", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:16 asynchttp.go:292: Completed job a1ee9108850351dfccbc52b87ed7b59a in 1.918652367s >[negroni] Started GET /queue/a1ee9108850351dfccbc52b87ed7b59a >[negroni] Completed 303 See Other in 148.992µs >[negroni] Started GET /blockvolumes/c09c849b291f015f8d35bdc68e9dca65 >[negroni] Completed 200 OK in 2.515816ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 20.637941ms >[asynchttp] INFO 2018/08/19 13:10:17 asynchttp.go:288: Started job f524339f1742b47fa2146bc53629f11d >[heketi] INFO 2018/08/19 13:10:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/f524339f1742b47fa2146bc53629f11d >[negroni] Completed 200 OK in 144.353µs >[cmdexec] INFO 2018/08/19 13:10:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f524339f1742b47fa2146bc53629f11d >[negroni] Completed 200 OK in 189.797µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 19.836008ms >[asynchttp] INFO 2018/08/19 13:10:19 asynchttp.go:288: Started job 7fbab5fae9828501ac8f763973606fef >[heketi] INFO 2018/08/19 13:10:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/7fbab5fae9828501ac8f763973606fef >[negroni] Completed 200 OK in 117.371µs >[cmdexec] INFO 2018/08/19 13:10:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f524339f1742b47fa2146bc53629f11d >[negroni] Completed 200 OK in 141.646µs >[kubeexec] DEBUG 2018/08/19 13:10:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_7dfc90d03c7e431e4c725023a4e210ba ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:dac66593-9727-478d-bc05-8d1c3632cbe1", "USERNAME": "dac66593-9727-478d-bc05-8d1c3632cbe1", "PASSWORD": "34d9383f-ace6-46cc-a7ff-5e1b0e9f67d4", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:19 asynchttp.go:292: Completed job f524339f1742b47fa2146bc53629f11d in 2.072565899s >[kubeexec] DEBUG 2018/08/19 13:10:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7fbab5fae9828501ac8f763973606fef >[negroni] Completed 200 OK in 185.567µs >[negroni] Started GET /queue/f524339f1742b47fa2146bc53629f11d >[negroni] Completed 303 See Other in 152.324µs >[negroni] Started GET /blockvolumes/7dfc90d03c7e431e4c725023a4e210ba >[negroni] Completed 200 OK in 2.01783ms >[negroni] Started GET /queue/7fbab5fae9828501ac8f763973606fef >[negroni] Completed 200 OK in 156.868µs >[kubeexec] DEBUG 2018/08/19 13:10:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_243f8e62a2cc230dacfeeaabec726757 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fa143f8a-5987-4f8d-869d-1b377bbe614c", "USERNAME": "fa143f8a-5987-4f8d-869d-1b377bbe614c", "PASSWORD": "dcc10f65-1bf5-45f8-8b11-43f10ac5f8b6", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:21 asynchttp.go:292: Completed job 7fbab5fae9828501ac8f763973606fef in 2.192778887s >[negroni] Started GET /queue/7fbab5fae9828501ac8f763973606fef >[negroni] Completed 303 See Other in 130.712µs >[negroni] Started GET /blockvolumes/243f8e62a2cc230dacfeeaabec726757 >[negroni] Completed 200 OK in 2.76141ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 19.874017ms >[asynchttp] INFO 2018/08/19 13:10:22 asynchttp.go:288: Started job 47c11572002b716c6d3e166d64334b20 >[heketi] INFO 2018/08/19 13:10:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/47c11572002b716c6d3e166d64334b20 >[negroni] Completed 200 OK in 103.699µs >[cmdexec] INFO 2018/08/19 13:10:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/47c11572002b716c6d3e166d64334b20 >[negroni] Completed 200 OK in 127.724µs >[negroni] Started GET /queue/47c11572002b716c6d3e166d64334b20 >[negroni] Completed 200 OK in 192.678µs >[kubeexec] DEBUG 2018/08/19 13:10:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_8bd71a3271607ce8ab6d6f723ed024f7 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:211e6a74-8f08-401a-9373-bee138c5d88a", "USERNAME": "211e6a74-8f08-401a-9373-bee138c5d88a", "PASSWORD": "35c28bcb-3658-456a-b6ff-6be1fa2546f8", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:25 asynchttp.go:292: Completed job 47c11572002b716c6d3e166d64334b20 in 2.170458529s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 36.646349ms >[asynchttp] INFO 2018/08/19 13:10:25 asynchttp.go:288: Started job de0dabc950c335101cd43f3d6cb52fcc >[heketi] INFO 2018/08/19 13:10:25 Started async operation: Create Block Volume >[negroni] Started GET /queue/de0dabc950c335101cd43f3d6cb52fcc >[negroni] Completed 200 OK in 114.705µs >[cmdexec] INFO 2018/08/19 13:10:25 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/47c11572002b716c6d3e166d64334b20 >[negroni] Completed 303 See Other in 137.519µs >[negroni] Started GET /blockvolumes/8bd71a3271607ce8ab6d6f723ed024f7 >[negroni] Completed 200 OK in 1.583232ms >[kubeexec] DEBUG 2018/08/19 13:10:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:26 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/de0dabc950c335101cd43f3d6cb52fcc >[negroni] Completed 200 OK in 217.986µs >[negroni] Started GET /queue/de0dabc950c335101cd43f3d6cb52fcc >[negroni] Completed 200 OK in 166.9µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 22.928043ms >[asynchttp] INFO 2018/08/19 13:10:28 asynchttp.go:288: Started job 4b48f0d5e0b9e1db9331f69e2aa89da3 >[heketi] INFO 2018/08/19 13:10:28 Started async operation: Create Block Volume >[negroni] Started GET /queue/4b48f0d5e0b9e1db9331f69e2aa89da3 >[negroni] Completed 200 OK in 138.764µs >[cmdexec] INFO 2018/08/19 13:10:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_944f3904db7463d2d05db63147822cb9 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fd33f36a-a2c5-46d0-a7aa-b7bb5902d4bb", "USERNAME": "fd33f36a-a2c5-46d0-a7aa-b7bb5902d4bb", "PASSWORD": "28d8ce75-d0ba-45a7-8d4c-4aa7dc91a334", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:28 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:28 asynchttp.go:292: Completed job de0dabc950c335101cd43f3d6cb52fcc in 2.928967215s >[kubeexec] DEBUG 2018/08/19 13:10:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:28 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/de0dabc950c335101cd43f3d6cb52fcc >[negroni] Completed 303 See Other in 192.29µs >[negroni] Started GET /blockvolumes/944f3904db7463d2d05db63147822cb9 >[negroni] Completed 200 OK in 1.696072ms >[kubeexec] DEBUG 2018/08/19 13:10:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4b48f0d5e0b9e1db9331f69e2aa89da3 >[negroni] Completed 200 OK in 142.269µs >[negroni] Started GET /queue/4b48f0d5e0b9e1db9331f69e2aa89da3 >[negroni] Completed 200 OK in 188.223µs >[kubeexec] DEBUG 2018/08/19 13:10:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_2d70841563ead792679a950debf90e7c ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f4aad26c-7e40-4469-8c1d-0d778ce84cd4", "USERNAME": "f4aad26c-7e40-4469-8c1d-0d778ce84cd4", "PASSWORD": "ebfa2711-4a47-47ff-a95e-7dfbc637b5ad", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:30 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:30 asynchttp.go:292: Completed job 4b48f0d5e0b9e1db9331f69e2aa89da3 in 2.340402764s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 33.230583ms >[asynchttp] INFO 2018/08/19 13:10:31 asynchttp.go:288: Started job 4f63d0b04d85a2f7041b8b5a8dd9624c >[heketi] INFO 2018/08/19 13:10:31 Started async operation: Create Block Volume >[negroni] Started GET /queue/4f63d0b04d85a2f7041b8b5a8dd9624c >[negroni] Completed 200 OK in 168.589µs >[cmdexec] INFO 2018/08/19 13:10:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:31 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4b48f0d5e0b9e1db9331f69e2aa89da3 >[negroni] Completed 303 See Other in 98.571µs >[negroni] Started GET /blockvolumes/2d70841563ead792679a950debf90e7c >[negroni] Completed 200 OK in 1.578034ms >[kubeexec] DEBUG 2018/08/19 13:10:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:31 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4f63d0b04d85a2f7041b8b5a8dd9624c >[negroni] Completed 200 OK in 135.152µs >[negroni] Started GET /queue/4f63d0b04d85a2f7041b8b5a8dd9624c >[negroni] Completed 200 OK in 173.702µs >[kubeexec] DEBUG 2018/08/19 13:10:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_e28ded43ddf2e54f36748657a3d294ae ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6aa95045-1159-4394-a146-3a2490726938", "USERNAME": "6aa95045-1159-4394-a146-3a2490726938", "PASSWORD": "0f522e62-cc1a-4ba1-b7e1-d57759d76977", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:33 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:33 asynchttp.go:292: Completed job 4f63d0b04d85a2f7041b8b5a8dd9624c in 2.406291995s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 82.265718ms >[asynchttp] INFO 2018/08/19 13:10:34 asynchttp.go:288: Started job 63565210507c4658721e4b495d3a5969 >[heketi] INFO 2018/08/19 13:10:34 Started async operation: Create Block Volume >[negroni] Started GET /queue/63565210507c4658721e4b495d3a5969 >[negroni] Completed 200 OK in 189.648µs >[cmdexec] INFO 2018/08/19 13:10:34 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4f63d0b04d85a2f7041b8b5a8dd9624c >[negroni] Completed 303 See Other in 219.053µs >[negroni] Started GET /blockvolumes/e28ded43ddf2e54f36748657a3d294ae >[negroni] Completed 200 OK in 1.880861ms >[kubeexec] DEBUG 2018/08/19 13:10:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:34 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/63565210507c4658721e4b495d3a5969 >[negroni] Completed 200 OK in 166.858µs >[negroni] Started GET /queue/63565210507c4658721e4b495d3a5969 >[negroni] Completed 200 OK in 170.325µs >[heketi] INFO 2018/08/19 13:10:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:10:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_98323653504a36b46dd658f09c7d24f6 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:9a6d1df1-1318-440a-8ca1-fa336313a65b", "USERNAME": "9a6d1df1-1318-440a-8ca1-fa336313a65b", "PASSWORD": "6bb0048f-bbbb-417d-86da-bfd20e194879", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[negroni] Started POST /blockvolumes >[negroni] Started GET /queue/63565210507c4658721e4b495d3a5969 >[negroni] Completed 200 OK in 206.452µs >[heketi] INFO 2018/08/19 13:10:37 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:37 asynchttp.go:292: Completed job 63565210507c4658721e4b495d3a5969 in 3.176655271s >[negroni] Completed 202 Accepted in 266.023264ms >[asynchttp] INFO 2018/08/19 13:10:37 asynchttp.go:288: Started job 71a7a859807de7389e4569ee2ecd101a >[heketi] INFO 2018/08/19 13:10:37 Started async operation: Create Block Volume >[negroni] Started GET /queue/71a7a859807de7389e4569ee2ecd101a >[negroni] Completed 200 OK in 181.503µs >[cmdexec] INFO 2018/08/19 13:10:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:10:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:10:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:10:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:10:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:10:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:10:37 Cleaned 0 nodes from health cache >[kubeexec] DEBUG 2018/08/19 13:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/63565210507c4658721e4b495d3a5969 >[negroni] Completed 303 See Other in 179.013µs >[negroni] Started GET /blockvolumes/98323653504a36b46dd658f09c7d24f6 >[negroni] Completed 200 OK in 1.595822ms >[negroni] Started GET /queue/71a7a859807de7389e4569ee2ecd101a >[negroni] Completed 200 OK in 296.078µs >[kubeexec] DEBUG 2018/08/19 13:10:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/71a7a859807de7389e4569ee2ecd101a >[negroni] Completed 200 OK in 186.517µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 40.004786ms >[asynchttp] INFO 2018/08/19 13:10:40 asynchttp.go:288: Started job 4168fa4a77f8e38a9f1007131e62dc36 >[heketi] INFO 2018/08/19 13:10:40 Started async operation: Create Block Volume >[negroni] Started GET /queue/4168fa4a77f8e38a9f1007131e62dc36 >[negroni] Completed 200 OK in 145.168µs >[cmdexec] INFO 2018/08/19 13:10:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_ea968b89cd92a3ae1c6087bb638dd0d1 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f543e5b6-63d8-4607-96d7-7134d4a29ad0", "USERNAME": "f543e5b6-63d8-4607-96d7-7134d4a29ad0", "PASSWORD": "9a96b367-11fc-410e-ac7a-73064b7bae19", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/71a7a859807de7389e4569ee2ecd101a >[negroni] Completed 200 OK in 222.753µs >[heketi] INFO 2018/08/19 13:10:40 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:40 asynchttp.go:292: Completed job 71a7a859807de7389e4569ee2ecd101a in 3.019980633s >[kubeexec] DEBUG 2018/08/19 13:10:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4168fa4a77f8e38a9f1007131e62dc36 >[negroni] Completed 200 OK in 176.759µs >[negroni] Started GET /queue/71a7a859807de7389e4569ee2ecd101a >[negroni] Completed 303 See Other in 154.788µs >[negroni] Started GET /blockvolumes/ea968b89cd92a3ae1c6087bb638dd0d1 >[negroni] Completed 200 OK in 1.815806ms >[negroni] Started GET /queue/4168fa4a77f8e38a9f1007131e62dc36 >[negroni] Completed 200 OK in 186.483µs >[kubeexec] DEBUG 2018/08/19 13:10:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_d2f806026ec0661933c525a22b9051ea ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:80004a9e-ed29-4082-bb13-8690ab1dd354", "USERNAME": "80004a9e-ed29-4082-bb13-8690ab1dd354", "PASSWORD": "e02898a3-2418-4e23-aaf1-450ca3a9bb9e", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:43 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:43 asynchttp.go:292: Completed job 4168fa4a77f8e38a9f1007131e62dc36 in 2.874035138s >[negroni] Started GET /queue/4168fa4a77f8e38a9f1007131e62dc36 >[negroni] Completed 303 See Other in 179.917µs >[negroni] Started GET /blockvolumes/d2f806026ec0661933c525a22b9051ea >[negroni] Completed 200 OK in 2.926454ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 37.386965ms >[asynchttp] INFO 2018/08/19 13:10:43 asynchttp.go:288: Started job 7a91590f96c576516606f6332eb0f9f7 >[heketi] INFO 2018/08/19 13:10:43 Started async operation: Create Block Volume >[negroni] Started GET /queue/7a91590f96c576516606f6332eb0f9f7 >[negroni] Completed 200 OK in 169.149µs >[cmdexec] INFO 2018/08/19 13:10:43 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:43 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7a91590f96c576516606f6332eb0f9f7 >[negroni] Completed 200 OK in 117.819µs >[negroni] Started GET /queue/7a91590f96c576516606f6332eb0f9f7 >[negroni] Completed 200 OK in 209.555µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 31.096959ms >[asynchttp] INFO 2018/08/19 13:10:46 asynchttp.go:288: Started job 885d627cd27fc0f9d5b1c7a60c9e34f7 >[heketi] INFO 2018/08/19 13:10:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/885d627cd27fc0f9d5b1c7a60c9e34f7 >[negroni] Completed 200 OK in 129.942µs >[cmdexec] INFO 2018/08/19 13:10:46 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1de17d8acd55d3d5d23ca73d2215a44c ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f94f052a-1a17-4fa1-a05d-19fb4c23cc9e", "USERNAME": "f94f052a-1a17-4fa1-a05d-19fb4c23cc9e", "PASSWORD": "c43e5340-f2c1-4b68-b3cf-ae838ec9c4c3", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:46 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:46 asynchttp.go:292: Completed job 7a91590f96c576516606f6332eb0f9f7 in 2.886131785s >[kubeexec] DEBUG 2018/08/19 13:10:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7a91590f96c576516606f6332eb0f9f7 >[negroni] Completed 303 See Other in 94.769µs >[negroni] Started GET /blockvolumes/1de17d8acd55d3d5d23ca73d2215a44c >[negroni] Completed 200 OK in 2.242881ms >[kubeexec] DEBUG 2018/08/19 13:10:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/885d627cd27fc0f9d5b1c7a60c9e34f7 >[negroni] Completed 200 OK in 129.34µs >[negroni] Started GET /queue/885d627cd27fc0f9d5b1c7a60c9e34f7 >[negroni] Completed 200 OK in 147.096µs >[kubeexec] DEBUG 2018/08/19 13:10:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_3541516cb4259f4f0f8848ce5b3c842a ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:28d1cfa6-be37-4df2-a90e-79319978eb18", "USERNAME": "28d1cfa6-be37-4df2-a90e-79319978eb18", "PASSWORD": "58f3427c-cf23-452e-a6fe-d3bd52474db0", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:48 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:48 asynchttp.go:292: Completed job 885d627cd27fc0f9d5b1c7a60c9e34f7 in 2.66496292s >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 132.687971ms >[asynchttp] INFO 2018/08/19 13:10:49 asynchttp.go:288: Started job 7d191d588a04daa97010bbda7cc85221 >[heketi] INFO 2018/08/19 13:10:49 Started async operation: Create Block Volume >[negroni] Started GET /queue/7d191d588a04daa97010bbda7cc85221 >[negroni] Completed 200 OK in 154.907µs >[cmdexec] INFO 2018/08/19 13:10:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/885d627cd27fc0f9d5b1c7a60c9e34f7 >[negroni] Completed 303 See Other in 214.151µs >[negroni] Started GET /blockvolumes/3541516cb4259f4f0f8848ce5b3c842a >[negroni] Completed 200 OK in 1.74658ms >[kubeexec] DEBUG 2018/08/19 13:10:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7d191d588a04daa97010bbda7cc85221 >[negroni] Completed 200 OK in 145.673µs >[negroni] Started GET /queue/7d191d588a04daa97010bbda7cc85221 >[negroni] Completed 200 OK in 304.212µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.209824ms >[asynchttp] INFO 2018/08/19 13:10:51 asynchttp.go:288: Started job c1feded05100868aa18f9f12d07900d1 >[heketi] INFO 2018/08/19 13:10:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/c1feded05100868aa18f9f12d07900d1 >[negroni] Completed 200 OK in 100.97µs >[cmdexec] INFO 2018/08/19 13:10:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_e9440f558e3c93a454b2aa30b616abd4 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:be419480-ffe0-4cc5-94b7-267d59f47c32", "USERNAME": "be419480-ffe0-4cc5-94b7-267d59f47c32", "PASSWORD": "e878d6ce-e18a-4d9a-b660-58e84e1ac56c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:51 asynchttp.go:292: Completed job 7d191d588a04daa97010bbda7cc85221 in 2.892047179s >[kubeexec] DEBUG 2018/08/19 13:10:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7d191d588a04daa97010bbda7cc85221 >[negroni] Completed 303 See Other in 160.414µs >[negroni] Started GET /blockvolumes/e9440f558e3c93a454b2aa30b616abd4 >[negroni] Completed 200 OK in 2.541393ms >[kubeexec] DEBUG 2018/08/19 13:10:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c1feded05100868aa18f9f12d07900d1 >[negroni] Completed 200 OK in 155.593µs >[kubeexec] DEBUG 2018/08/19 13:10:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c1feded05100868aa18f9f12d07900d1 >[negroni] Completed 200 OK in 224.579µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 29.267028ms >[asynchttp] INFO 2018/08/19 13:10:54 asynchttp.go:288: Started job c3f39ced53dc5f7813e70c32a98a4025 >[heketi] INFO 2018/08/19 13:10:54 Started async operation: Create Block Volume >[negroni] Started GET /queue/c3f39ced53dc5f7813e70c32a98a4025 >[negroni] Completed 200 OK in 115.071µs >[cmdexec] INFO 2018/08/19 13:10:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c1feded05100868aa18f9f12d07900d1 >[negroni] Completed 200 OK in 235.619µs >[kubeexec] DEBUG 2018/08/19 13:10:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_bd4d84175587d8d57496d6f85c450e27 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:46e9ee63-4952-4d8e-91a7-45fea5046992", "USERNAME": "46e9ee63-4952-4d8e-91a7-45fea5046992", "PASSWORD": "ba945074-1dd7-4ea8-b967-4269ce99f4ad", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:54 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:54 asynchttp.go:292: Completed job c1feded05100868aa18f9f12d07900d1 in 3.109403919s >[kubeexec] DEBUG 2018/08/19 13:10:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:54 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c3f39ced53dc5f7813e70c32a98a4025 >[negroni] Completed 200 OK in 258.098µs >[negroni] Started GET /queue/c1feded05100868aa18f9f12d07900d1 >[negroni] Completed 303 See Other in 170.23µs >[negroni] Started GET /blockvolumes/bd4d84175587d8d57496d6f85c450e27 >[negroni] Completed 200 OK in 2.038327ms >[kubeexec] DEBUG 2018/08/19 13:10:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c3f39ced53dc5f7813e70c32a98a4025 >[negroni] Completed 200 OK in 136.871µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 26.843299ms >[asynchttp] INFO 2018/08/19 13:10:57 asynchttp.go:288: Started job 42466e79b680284e6dd68db001a3e670 >[heketi] INFO 2018/08/19 13:10:57 Started async operation: Create Block Volume >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 200 OK in 137.266µs >[cmdexec] INFO 2018/08/19 13:10:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c3f39ced53dc5f7813e70c32a98a4025 >[negroni] Completed 200 OK in 164.6µs >[kubeexec] DEBUG 2018/08/19 13:10:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_f77beb6f718b0fa8153e43f5a2b8ab66 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:61e5bdbd-efdd-4703-8c3e-8e6d5ded0bea", "USERNAME": "61e5bdbd-efdd-4703-8c3e-8e6d5ded0bea", "PASSWORD": "6a18bc96-9abb-46ea-8766-6582e3a62563", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:10:58 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:10:58 asynchttp.go:292: Completed job c3f39ced53dc5f7813e70c32a98a4025 in 3.549219656s >[kubeexec] DEBUG 2018/08/19 13:10:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:10:58 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 200 OK in 186.599µs >[negroni] Started GET /queue/c3f39ced53dc5f7813e70c32a98a4025 >[negroni] Completed 303 See Other in 174.888µs >[negroni] Started GET /blockvolumes/f77beb6f718b0fa8153e43f5a2b8ab66 >[negroni] Completed 200 OK in 2.813557ms >[kubeexec] DEBUG 2018/08/19 13:10:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:10:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:10:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:10:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 200 OK in 126.854µs >[kubeexec] DEBUG 2018/08/19 13:10:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 25.378406ms >[asynchttp] INFO 2018/08/19 13:11:00 asynchttp.go:288: Started job 33ef2813cc925fafeabe4ceac62263f9 >[heketi] INFO 2018/08/19 13:11:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 200 OK in 152.453µs >[cmdexec] INFO 2018/08/19 13:11:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 200 OK in 111.613µs >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 200 OK in 121.146µs >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 200 OK in 149.749µs >[kubeexec] DEBUG 2018/08/19 13:11:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6a815acd2c41bdf0364b8ca43b391e94 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c1f4c1e5-989e-48d5-98f9-412df94a42de", "USERNAME": "c1f4c1e5-989e-48d5-98f9-412df94a42de", "PASSWORD": "cef560ca-1463-4dcf-a71b-ea5f97257bad", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:01 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:01 asynchttp.go:292: Completed job 42466e79b680284e6dd68db001a3e670 in 4.441788455s >[kubeexec] DEBUG 2018/08/19 13:11:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 200 OK in 117.199µs >[kubeexec] DEBUG 2018/08/19 13:11:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:02 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/42466e79b680284e6dd68db001a3e670 >[negroni] Completed 303 See Other in 130.716µs >[negroni] Started GET /blockvolumes/6a815acd2c41bdf0364b8ca43b391e94 >[negroni] Completed 200 OK in 2.797763ms >[kubeexec] DEBUG 2018/08/19 13:11:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 200 OK in 207.464µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 184.458165ms >[asynchttp] INFO 2018/08/19 13:11:03 asynchttp.go:288: Started job 7e3b3f98b550579cb6d4e49acede4aec >[heketi] INFO 2018/08/19 13:11:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/7e3b3f98b550579cb6d4e49acede4aec >[negroni] Completed 200 OK in 137.172µs >[cmdexec] INFO 2018/08/19 13:11:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 200 OK in 171.947µs >[kubeexec] DEBUG 2018/08/19 13:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:04 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1e5fafb98450b38ad941d9aec6baeabc ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:daecffd9-660b-4c91-aa67-51ef8c49b4bc", "USERNAME": "daecffd9-660b-4c91-aa67-51ef8c49b4bc", "PASSWORD": "02494018-9b04-44f0-afd9-3c2abcc13d9d", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:04 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:04 asynchttp.go:292: Completed job 33ef2813cc925fafeabe4ceac62263f9 in 4.721801693s >[negroni] Started GET /queue/7e3b3f98b550579cb6d4e49acede4aec >[negroni] Completed 200 OK in 158.422µs >[kubeexec] DEBUG 2018/08/19 13:11:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/33ef2813cc925fafeabe4ceac62263f9 >[negroni] Completed 303 See Other in 154.038µs >[negroni] Started GET /blockvolumes/1e5fafb98450b38ad941d9aec6baeabc >[negroni] Completed 200 OK in 1.912724ms >[negroni] Started GET /queue/7e3b3f98b550579cb6d4e49acede4aec >[negroni] Completed 200 OK in 148.537µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.385197ms >[asynchttp] INFO 2018/08/19 13:11:06 asynchttp.go:288: Started job 9c213e8ec851a6a39e17c9d34a293e62 >[heketi] INFO 2018/08/19 13:11:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 200 OK in 107.494µs >[cmdexec] INFO 2018/08/19 13:11:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7e3b3f98b550579cb6d4e49acede4aec >[negroni] Completed 200 OK in 206.549µs >[kubeexec] DEBUG 2018/08/19 13:11:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_e761e65d45d87d7912aeb979c565a187 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4b8f068f-fd5f-42b2-99aa-8e346024a574", "USERNAME": "4b8f068f-fd5f-42b2-99aa-8e346024a574", "PASSWORD": "23c25eef-5112-477b-8ed0-88e0bac5d17b", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:07 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:07 asynchttp.go:292: Completed job 7e3b3f98b550579cb6d4e49acede4aec in 3.337811382s >[kubeexec] DEBUG 2018/08/19 13:11:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:07 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 200 OK in 152.378µs >[kubeexec] DEBUG 2018/08/19 13:11:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7e3b3f98b550579cb6d4e49acede4aec >[negroni] Completed 303 See Other in 132.796µs >[negroni] Started GET /blockvolumes/e761e65d45d87d7912aeb979c565a187 >[negroni] Completed 200 OK in 1.774652ms >[kubeexec] DEBUG 2018/08/19 13:11:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 200 OK in 165.441µs >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 200 OK in 192.626µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 36.693339ms >[asynchttp] INFO 2018/08/19 13:11:10 asynchttp.go:288: Started job 19cd59c276be06c9bac2afd9b98318c8 >[heketi] INFO 2018/08/19 13:11:10 Started async operation: Create Block Volume >[negroni] Started GET /queue/19cd59c276be06c9bac2afd9b98318c8 >[negroni] Completed 200 OK in 143.313µs >[cmdexec] INFO 2018/08/19 13:11:10 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 200 OK in 159.897µs >[kubeexec] DEBUG 2018/08/19 13:11:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c9fa470dd86ea031a1aaa5a0eeac5222 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4c8c96c1-6761-40c5-82a6-f945c54d1179", "USERNAME": "4c8c96c1-6761-40c5-82a6-f945c54d1179", "PASSWORD": "b0c20492-e5e2-49f7-be53-a3b12236aa0e", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:10 asynchttp.go:292: Completed job 9c213e8ec851a6a39e17c9d34a293e62 in 4.247389649s >[kubeexec] DEBUG 2018/08/19 13:11:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:10 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/19cd59c276be06c9bac2afd9b98318c8 >[negroni] Completed 200 OK in 165.557µs >[negroni] Started GET /queue/9c213e8ec851a6a39e17c9d34a293e62 >[negroni] Completed 303 See Other in 196.823µs >[negroni] Started GET /blockvolumes/c9fa470dd86ea031a1aaa5a0eeac5222 >[negroni] Completed 200 OK in 1.95636ms >[negroni] Started GET /queue/19cd59c276be06c9bac2afd9b98318c8 >[negroni] Completed 200 OK in 145.473µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 27.959567ms >[asynchttp] INFO 2018/08/19 13:11:12 asynchttp.go:288: Started job 17a6cb409153099de7f2d833b58b50ff >[heketi] INFO 2018/08/19 13:11:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/17a6cb409153099de7f2d833b58b50ff >[negroni] Completed 200 OK in 106.737µs >[cmdexec] INFO 2018/08/19 13:11:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_03244e5daef9d7b2a3a8a99152fa08a6 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:99026c09-997d-4ed8-a23a-f2f6aba91472", "USERNAME": "99026c09-997d-4ed8-a23a-f2f6aba91472", "PASSWORD": "64e9eb2c-8f23-4aa3-9581-eb9be514d59c", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/19cd59c276be06c9bac2afd9b98318c8 >[negroni] Completed 200 OK in 135.394µs >[heketi] INFO 2018/08/19 13:11:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:13 asynchttp.go:292: Completed job 19cd59c276be06c9bac2afd9b98318c8 in 3.027300503s >[negroni] Started GET /queue/17a6cb409153099de7f2d833b58b50ff >[negroni] Completed 200 OK in 151.777µs >[kubeexec] DEBUG 2018/08/19 13:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/19cd59c276be06c9bac2afd9b98318c8 >[negroni] Completed 303 See Other in 194.595µs >[negroni] Started GET /blockvolumes/03244e5daef9d7b2a3a8a99152fa08a6 >[negroni] Completed 200 OK in 1.656719ms >[negroni] Started GET /queue/17a6cb409153099de7f2d833b58b50ff >[negroni] Completed 200 OK in 194.359µs >[negroni] Started GET /queue/17a6cb409153099de7f2d833b58b50ff >[negroni] Completed 200 OK in 132.886µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.785554ms >[asynchttp] INFO 2018/08/19 13:11:15 asynchttp.go:288: Started job 62d6affb9d922026d83fe65b1b76a8e0 >[heketi] INFO 2018/08/19 13:11:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/62d6affb9d922026d83fe65b1b76a8e0 >[negroni] Completed 200 OK in 127.792µs >[cmdexec] INFO 2018/08/19 13:11:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_7df006abc8f13b877e9006417059ec0e ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4f3b2b41-e8f0-4799-9673-8887ee436df3", "USERNAME": "4f3b2b41-e8f0-4799-9673-8887ee436df3", "PASSWORD": "32f05926-0e27-426b-ba9f-a9ee14be3885", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:16 asynchttp.go:292: Completed job 17a6cb409153099de7f2d833b58b50ff in 3.7386492s >[kubeexec] DEBUG 2018/08/19 13:11:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/17a6cb409153099de7f2d833b58b50ff >[negroni] Completed 303 See Other in 277.04µs >[negroni] Started GET /blockvolumes/7df006abc8f13b877e9006417059ec0e >[negroni] Completed 200 OK in 1.570038ms >[negroni] Started GET /queue/62d6affb9d922026d83fe65b1b76a8e0 >[negroni] Completed 200 OK in 177.508µs >[kubeexec] DEBUG 2018/08/19 13:11:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/62d6affb9d922026d83fe65b1b76a8e0 >[negroni] Completed 200 OK in 192.839µs >[negroni] Started GET /queue/62d6affb9d922026d83fe65b1b76a8e0 >[negroni] Completed 200 OK in 158.982µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.324939ms >[asynchttp] INFO 2018/08/19 13:11:18 asynchttp.go:288: Started job 4ed1de0b33b7cdcfcad4bb8b868206e0 >[heketi] INFO 2018/08/19 13:11:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/4ed1de0b33b7cdcfcad4bb8b868206e0 >[negroni] Completed 200 OK in 129.306µs >[cmdexec] INFO 2018/08/19 13:11:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_520b7dd900f44aec2c29d3c35649662b ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:1cab4993-447a-47a5-9cbb-2e02be5479c9", "USERNAME": "1cab4993-447a-47a5-9cbb-2e02be5479c9", "PASSWORD": "22dd2bc6-9cb3-45a5-833b-15087bff4989", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:19 asynchttp.go:292: Completed job 62d6affb9d922026d83fe65b1b76a8e0 in 3.348158459s >[kubeexec] DEBUG 2018/08/19 13:11:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 15min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:11:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:11:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:11:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:11:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/62d6affb9d922026d83fe65b1b76a8e0 >[negroni] Completed 303 See Other in 168.627µs >[negroni] Started GET /blockvolumes/520b7dd900f44aec2c29d3c35649662b >[negroni] Completed 200 OK in 2.361855ms >[negroni] Started GET /queue/4ed1de0b33b7cdcfcad4bb8b868206e0 >[negroni] Completed 200 OK in 122.967µs >[kubeexec] DEBUG 2018/08/19 13:11:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4ed1de0b33b7cdcfcad4bb8b868206e0 >[negroni] Completed 200 OK in 173.435µs >[negroni] Started GET /queue/4ed1de0b33b7cdcfcad4bb8b868206e0 >[negroni] Completed 200 OK in 119.123µs >[kubeexec] DEBUG 2018/08/19 13:11:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_bf5549ef3f43d689c226f74e183f09c0 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a4c764e0-fe93-4b55-a363-8a3c91b05c46", "USERNAME": "a4c764e0-fe93-4b55-a363-8a3c91b05c46", "PASSWORD": "ca3679ad-a4bc-46b8-900e-fa17b6474cbf", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:11:22 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:11:22 asynchttp.go:292: Completed job 4ed1de0b33b7cdcfcad4bb8b868206e0 in 3.387881639s >[negroni] Started GET /queue/4ed1de0b33b7cdcfcad4bb8b868206e0 >[negroni] Completed 303 See Other in 144.281µs >[negroni] Started GET /blockvolumes/bf5549ef3f43d689c226f74e183f09c0 >[negroni] Completed 200 OK in 2.428321ms >[heketi] INFO 2018/08/19 13:12:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:12:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:12:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:12:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:12:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:12:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:12:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:12:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:14:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:14:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:14:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:14:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:14:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:14:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:14:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:14:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:16:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:16:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:16:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:16:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:16:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:16:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:16:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:16:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:18:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:18:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:18:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:18:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:18:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:18:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:18:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:18:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 35.562117ms >[asynchttp] INFO 2018/08/19 13:19:20 asynchttp.go:288: Started job 9ddff910e1c840b6a675a7cadb0c8564 >[heketi] INFO 2018/08/19 13:19:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 188.819µs >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Completed 202 Accepted in 49.431768ms >[asynchttp] INFO 2018/08/19 13:19:20 asynchttp.go:288: Started job 4c9e0a475019fff4fbbe684027f05a19 >[heketi] INFO 2018/08/19 13:19:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/4c9e0a475019fff4fbbe684027f05a19 >[negroni] Completed 200 OK in 155.197µs >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 23min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 23min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:19:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:19:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:19:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 157.69µs >[negroni] Started GET /queue/4c9e0a475019fff4fbbe684027f05a19 >[negroni] Completed 200 OK in 165.439µs >[kubeexec] DEBUG 2018/08/19 13:19:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 157.342µs >[negroni] Started GET /queue/4c9e0a475019fff4fbbe684027f05a19 >[negroni] Completed 200 OK in 138.674µs >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 164.121µs >[negroni] Started GET /queue/4c9e0a475019fff4fbbe684027f05a19 >[negroni] Completed 200 OK in 141.847µs >[kubeexec] DEBUG 2018/08/19 13:19:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_a8336f8a16c4ed5e5c6a9c1baeadfff9 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d9d7e4e3-7e9a-4764-83f5-6f9e735e08d5", "USERNAME": "d9d7e4e3-7e9a-4764-83f5-6f9e735e08d5", "PASSWORD": "e84e0f3c-0240-4bc5-a69d-d6506069905c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:19:23 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:19:23 asynchttp.go:292: Completed job 4c9e0a475019fff4fbbe684027f05a19 in 3.48076385s >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 137.289µs >[negroni] Started GET /queue/4c9e0a475019fff4fbbe684027f05a19 >[negroni] Completed 303 See Other in 174.723µs >[negroni] Started GET /blockvolumes/a8336f8a16c4ed5e5c6a9c1baeadfff9 >[negroni] Completed 200 OK in 2.359633ms >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 200 OK in 124.151µs >[kubeexec] DEBUG 2018/08/19 13:19:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_d1677b2176855c0acefd268d2311b3fd ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a2556e08-d844-401e-844e-3673d96b5994", "USERNAME": "a2556e08-d844-401e-844e-3673d96b5994", "PASSWORD": "636d5d20-4bfc-4b59-9e21-e3fcf2af7200", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:19:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:19:25 asynchttp.go:292: Completed job 9ddff910e1c840b6a675a7cadb0c8564 in 5.472551707s >[negroni] Started GET /queue/9ddff910e1c840b6a675a7cadb0c8564 >[negroni] Completed 303 See Other in 125.222µs >[negroni] Started GET /blockvolumes/d1677b2176855c0acefd268d2311b3fd >[negroni] Completed 200 OK in 2.563874ms >[heketi] INFO 2018/08/19 13:20:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:20:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:20:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:20:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:20:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:20:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:20:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:20:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:22:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:22:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:22:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:22:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:22:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:22:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:22:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:22:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:24:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:24:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:24:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:24:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 28min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:24:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:24:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:24:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 22min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:24:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:24:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:26:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:26:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 30min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:26:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:26:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 30min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:26:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:26:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:26:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 24min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:26:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:26:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 37.685392ms >[asynchttp] INFO 2018/08/19 13:27:11 asynchttp.go:288: Started job 8f907955dd351426dd63d1fee1e5c159 >[heketi] INFO 2018/08/19 13:27:11 Started async operation: Create Block Volume >[negroni] Started GET /queue/8f907955dd351426dd63d1fee1e5c159 >[negroni] Completed 200 OK in 208.262µs >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Completed 202 Accepted in 64.019197ms >[asynchttp] INFO 2018/08/19 13:27:11 asynchttp.go:288: Started job 8849903aee5bc057744434494cf79d72 >[heketi] INFO 2018/08/19 13:27:11 Started async operation: Create Block Volume >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 109.469µs >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 30min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 31min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 30min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 31min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 24min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 24min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:27:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:27:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 31min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8f907955dd351426dd63d1fee1e5c159 >[negroni] Completed 200 OK in 169.638µs >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 142.353µs >[kubeexec] DEBUG 2018/08/19 13:27:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 31min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8f907955dd351426dd63d1fee1e5c159 >[negroni] Completed 200 OK in 181.271µs >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 139.545µs >[negroni] Started GET /queue/8f907955dd351426dd63d1fee1e5c159 >[negroni] Completed 200 OK in 149.661µs >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 132.725µs >[kubeexec] DEBUG 2018/08/19 13:27:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_4e5f56d76a5158fdcb747956e0ae5900 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:69d9cbd9-0663-488a-ae5c-89e51bf05575", "USERNAME": "69d9cbd9-0663-488a-ae5c-89e51bf05575", "PASSWORD": "c4453e39-e2c6-4e53-9e59-9d6512aabb1c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:27:15 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:27:15 asynchttp.go:292: Completed job 8f907955dd351426dd63d1fee1e5c159 in 3.858287494s >[negroni] Started GET /queue/8f907955dd351426dd63d1fee1e5c159 >[negroni] Completed 303 See Other in 134.168µs >[negroni] Started GET /blockvolumes/4e5f56d76a5158fdcb747956e0ae5900 >[negroni] Completed 200 OK in 2.748945ms >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 169.592µs >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 200 OK in 138.913µs >[kubeexec] DEBUG 2018/08/19 13:27:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_aa50d60b9d0b38dc013e903b12f069ce ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:19da5ebb-e588-467b-80c6-81a76c9f7ca9", "USERNAME": "19da5ebb-e588-467b-80c6-81a76c9f7ca9", "PASSWORD": "82d88a29-45f2-40e9-98e6-f11c0d7de778", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:27:17 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:27:17 asynchttp.go:292: Completed job 8849903aee5bc057744434494cf79d72 in 5.807728746s >[negroni] Started GET /queue/8849903aee5bc057744434494cf79d72 >[negroni] Completed 303 See Other in 236.606µs >[negroni] Started GET /blockvolumes/aa50d60b9d0b38dc013e903b12f069ce >[negroni] Completed 200 OK in 3.152439ms >[heketi] INFO 2018/08/19 13:28:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:28:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 32min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:28:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:28:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 32min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:28:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:28:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:28:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 26min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:28:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:28:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:30:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:30:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 34min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:30:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:30:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 34min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:30:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:30:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:30:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 28min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:30:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:30:37 Cleaned 0 nodes from health cache >[heketi] INFO 2018/08/19 13:32:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:32:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 36min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:32:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:32:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:32:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:32:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:32:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 30min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:32:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:32:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 63.317908ms >[asynchttp] INFO 2018/08/19 13:33:15 asynchttp.go:288: Started job 78bc9e13aa1391e5f6cf0c4fed1801f8 >[heketi] INFO 2018/08/19 13:33:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/78bc9e13aa1391e5f6cf0c4fed1801f8 >[negroni] Completed 200 OK in 127.95µs >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Completed 202 Accepted in 73.325811ms >[asynchttp] INFO 2018/08/19 13:33:15 asynchttp.go:288: Started job 69aa30dcdfc49e68b491da1ccad6c582 >[heketi] INFO 2018/08/19 13:33:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 111.783µs >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 30min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 30min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 36min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:33:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:33:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:33:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/78bc9e13aa1391e5f6cf0c4fed1801f8 >[negroni] Completed 200 OK in 177.227µs >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 114.767µs >[kubeexec] DEBUG 2018/08/19 13:33:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/78bc9e13aa1391e5f6cf0c4fed1801f8 >[negroni] Completed 200 OK in 168.565µs >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 109.716µs >[negroni] Started GET /queue/78bc9e13aa1391e5f6cf0c4fed1801f8 >[negroni] Completed 200 OK in 148.278µs >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 96.928µs >[kubeexec] DEBUG 2018/08/19 13:33:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6176f0ddeba61d6ca464b0ac8f230fe9 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b2841f8a-f34b-4718-8d1c-1cee7e38b7fb", "USERNAME": "b2841f8a-f34b-4718-8d1c-1cee7e38b7fb", "PASSWORD": "6f5e2afe-eb29-4106-9f95-50290a3634da", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:33:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:33:19 asynchttp.go:292: Completed job 78bc9e13aa1391e5f6cf0c4fed1801f8 in 3.539234422s >[negroni] Started GET /queue/78bc9e13aa1391e5f6cf0c4fed1801f8 >[negroni] Completed 303 See Other in 178.957µs >[negroni] Started GET /blockvolumes/6176f0ddeba61d6ca464b0ac8f230fe9 >[negroni] Completed 200 OK in 2.810438ms >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 117.427µs >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 200 OK in 144.992µs >[kubeexec] DEBUG 2018/08/19 13:33:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_2a49af4a9ece2876652cbe46d7d4511d ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6230c395-ad74-4d56-8287-edeb710931e3", "USERNAME": "6230c395-ad74-4d56-8287-edeb710931e3", "PASSWORD": "d2e80c45-6ef3-4288-a103-5deaa65837a3", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:33:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:33:21 asynchttp.go:292: Completed job 69aa30dcdfc49e68b491da1ccad6c582 in 5.654656219s >[negroni] Started GET /queue/69aa30dcdfc49e68b491da1ccad6c582 >[negroni] Completed 303 See Other in 132.631µs >[negroni] Started GET /blockvolumes/2a49af4a9ece2876652cbe46d7d4511d >[negroni] Completed 200 OK in 2.662432ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 37.015325ms >[asynchttp] INFO 2018/08/19 13:33:45 asynchttp.go:288: Started job 1fc7e81cd3f1e48e9bf1d0df583062fe >[heketi] INFO 2018/08/19 13:33:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/1fc7e81cd3f1e48e9bf1d0df583062fe >[negroni] Completed 200 OK in 136.21µs >[cmdexec] INFO 2018/08/19 13:33:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:33:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 31min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:33:46 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 37min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:33:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1fc7e81cd3f1e48e9bf1d0df583062fe >[negroni] Completed 200 OK in 212.279µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 22.994053ms >[asynchttp] INFO 2018/08/19 13:33:47 asynchttp.go:288: Started job 098a35a57a1e73b131616aa45e1c7131 >[heketi] INFO 2018/08/19 13:33:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 200 OK in 206.392µs >[cmdexec] INFO 2018/08/19 13:33:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 31min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:33:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:33:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 37min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:33:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1fc7e81cd3f1e48e9bf1d0df583062fe >[negroni] Completed 200 OK in 126.844µs >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 200 OK in 171.601µs >[kubeexec] DEBUG 2018/08/19 13:33:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_02b947dc5ddb7721492957f61780fdf2 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c9bc7f50-7b50-4e2e-91be-47830ab06a2d", "USERNAME": "c9bc7f50-7b50-4e2e-91be-47830ab06a2d", "PASSWORD": "76a7522d-7474-4258-bc36-bc2cd2ff410c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:33:48 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:33:48 asynchttp.go:292: Completed job 1fc7e81cd3f1e48e9bf1d0df583062fe in 2.826922158s >[kubeexec] DEBUG 2018/08/19 13:33:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:33:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1fc7e81cd3f1e48e9bf1d0df583062fe >[negroni] Completed 303 See Other in 119.867µs >[negroni] Started GET /blockvolumes/02b947dc5ddb7721492957f61780fdf2 >[negroni] Completed 200 OK in 1.488079ms >[kubeexec] DEBUG 2018/08/19 13:33:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 37min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 200 OK in 99.456µs >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 200 OK in 161.47µs >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 200 OK in 127.427µs >[kubeexec] DEBUG 2018/08/19 13:33:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_80169c94a0972218bdd55e998f554df5 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fffcc3f8-8a9d-42b3-8006-011d2aaab992", "USERNAME": "fffcc3f8-8a9d-42b3-8006-011d2aaab992", "PASSWORD": "140a076a-9504-49d7-aa0a-1ab2103af8a1", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:33:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:33:51 asynchttp.go:292: Completed job 098a35a57a1e73b131616aa45e1c7131 in 4.463553059s >[negroni] Started GET /queue/098a35a57a1e73b131616aa45e1c7131 >[negroni] Completed 303 See Other in 192.366µs >[negroni] Started GET /blockvolumes/80169c94a0972218bdd55e998f554df5 >[negroni] Completed 200 OK in 2.207097ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 27.367381ms >[asynchttp] INFO 2018/08/19 13:34:16 asynchttp.go:288: Started job 665e50f662cfad1ab96eed5fd8c24de9 >[heketi] INFO 2018/08/19 13:34:16 Started async operation: Create Block Volume >[negroni] Started GET /queue/665e50f662cfad1ab96eed5fd8c24de9 >[negroni] Completed 200 OK in 99.235µs >[cmdexec] INFO 2018/08/19 13:34:16 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 31min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:34:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:34:16 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 37min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:34:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/665e50f662cfad1ab96eed5fd8c24de9 >[negroni] Completed 200 OK in 161.696µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 29.372414ms >[asynchttp] INFO 2018/08/19 13:34:17 asynchttp.go:288: Started job 779ff0176e4952e8ba2074fc58fddc98 >[heketi] INFO 2018/08/19 13:34:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 200 OK in 104.629µs >[cmdexec] INFO 2018/08/19 13:34:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 31min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:34:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/665e50f662cfad1ab96eed5fd8c24de9 >[negroni] Completed 200 OK in 124.641µs >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 200 OK in 122.823µs >[kubeexec] DEBUG 2018/08/19 13:34:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_244b3f0cd3d35cb71240cf7eefe8dcd1 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ce256fb0-97c1-4374-96a8-9d2df97ca0b0", "USERNAME": "ce256fb0-97c1-4374-96a8-9d2df97ca0b0", "PASSWORD": "473a2f6e-7ed5-4e57-925d-283285ca24cb", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:34:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:34:19 asynchttp.go:292: Completed job 665e50f662cfad1ab96eed5fd8c24de9 in 2.86494573s >[kubeexec] DEBUG 2018/08/19 13:34:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:34:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/665e50f662cfad1ab96eed5fd8c24de9 >[negroni] Completed 303 See Other in 132.202µs >[negroni] Started GET /blockvolumes/244b3f0cd3d35cb71240cf7eefe8dcd1 >[negroni] Completed 200 OK in 2.062034ms >[kubeexec] DEBUG 2018/08/19 13:34:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:34:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 200 OK in 158.476µs >[kubeexec] DEBUG 2018/08/19 13:34:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 200 OK in 115.69µs >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 200 OK in 167.961µs >[kubeexec] DEBUG 2018/08/19 13:34:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c0a2727bcd6ac901124a3a0f9a8ad887 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:baea7304-e687-4b0e-9fc4-72dff40baf95", "USERNAME": "baea7304-e687-4b0e-9fc4-72dff40baf95", "PASSWORD": "4eb8f165-2ae6-4aeb-bb23-4575f444dcc3", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:34:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:34:21 asynchttp.go:292: Completed job 779ff0176e4952e8ba2074fc58fddc98 in 4.366452207s >[negroni] Started GET /queue/779ff0176e4952e8ba2074fc58fddc98 >[negroni] Completed 303 See Other in 201.823µs >[negroni] Started GET /blockvolumes/c0a2727bcd6ac901124a3a0f9a8ad887 >[negroni] Completed 200 OK in 2.962396ms >[heketi] INFO 2018/08/19 13:34:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:34:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:34:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:34:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:34:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:34:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 32min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:34:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:34:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 62.740883ms >[asynchttp] INFO 2018/08/19 13:34:46 asynchttp.go:288: Started job 18706ad8777a5f514ba4ac3833165e45 >[heketi] INFO 2018/08/19 13:34:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/18706ad8777a5f514ba4ac3833165e45 >[negroni] Completed 200 OK in 147.993µs >[cmdexec] INFO 2018/08/19 13:34:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 32min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:34:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:34:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:34:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/18706ad8777a5f514ba4ac3833165e45 >[negroni] Completed 200 OK in 125.096µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 24.346531ms >[asynchttp] INFO 2018/08/19 13:34:47 asynchttp.go:288: Started job 7293504a8d3e8403bb0e166547f7b6da >[heketi] INFO 2018/08/19 13:34:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 200 OK in 92.076µs >[cmdexec] INFO 2018/08/19 13:34:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/18706ad8777a5f514ba4ac3833165e45 >[negroni] Completed 200 OK in 120.732µs >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 200 OK in 119.431µs >[kubeexec] DEBUG 2018/08/19 13:34:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1bbb1d1de2fd57a8b59996a3e8daf8e2 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:be720b9d-7491-4d17-8f5e-572220d6532f", "USERNAME": "be720b9d-7491-4d17-8f5e-572220d6532f", "PASSWORD": "13e613e1-341b-46ff-b65e-15b5625f6576", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:34:49 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:34:49 asynchttp.go:292: Completed job 18706ad8777a5f514ba4ac3833165e45 in 2.89902488s >[negroni] Started GET /queue/18706ad8777a5f514ba4ac3833165e45 >[negroni] Completed 303 See Other in 143.234µs >[negroni] Started GET /blockvolumes/1bbb1d1de2fd57a8b59996a3e8daf8e2 >[negroni] Completed 200 OK in 2.796048ms >[kubeexec] DEBUG 2018/08/19 13:34:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:34:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:34:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 200 OK in 103.571µs >[kubeexec] DEBUG 2018/08/19 13:34:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 32min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:34:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:34:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 38min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 200 OK in 114.049µs >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 200 OK in 117.818µs >[kubeexec] DEBUG 2018/08/19 13:34:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1c0e677572fad0656e3a9a1735fd3cc5 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:cd124026-375a-4181-94d0-adbbe7c35aaa", "USERNAME": "cd124026-375a-4181-94d0-adbbe7c35aaa", "PASSWORD": "407a04e9-2777-4605-98a0-dd7b064e359f", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:34:52 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:34:52 asynchttp.go:292: Completed job 7293504a8d3e8403bb0e166547f7b6da in 4.604288534s >[negroni] Started GET /queue/7293504a8d3e8403bb0e166547f7b6da >[negroni] Completed 303 See Other in 126.261µs >[negroni] Started GET /blockvolumes/1c0e677572fad0656e3a9a1735fd3cc5 >[negroni] Completed 200 OK in 3.734277ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 39.686219ms >[asynchttp] INFO 2018/08/19 13:35:17 asynchttp.go:288: Started job dd2da7824e5b7a329f746647e71d126f >[heketi] INFO 2018/08/19 13:35:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/dd2da7824e5b7a329f746647e71d126f >[negroni] Completed 200 OK in 117.966µs >[cmdexec] INFO 2018/08/19 13:35:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:35:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 38min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:35:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 32min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:35:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/dd2da7824e5b7a329f746647e71d126f >[negroni] Completed 200 OK in 129.945µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.067357ms >[asynchttp] INFO 2018/08/19 13:35:18 asynchttp.go:288: Started job b7137a316645ccb9671431352495c604 >[heketi] INFO 2018/08/19 13:35:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 200 OK in 105.515µs >[cmdexec] INFO 2018/08/19 13:35:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/dd2da7824e5b7a329f746647e71d126f >[negroni] Completed 200 OK in 123.983µs >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 200 OK in 115.229µs >[kubeexec] DEBUG 2018/08/19 13:35:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_15731bca9a658eaf568941f44f58894b ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:dc5120ea-94bf-4fe3-9bc6-997acccacc5a", "USERNAME": "dc5120ea-94bf-4fe3-9bc6-997acccacc5a", "PASSWORD": "fa48d699-e602-4e3b-b2c1-9e46e2df9dd8", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:35:20 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:35:20 asynchttp.go:292: Completed job dd2da7824e5b7a329f746647e71d126f in 2.978524228s >[negroni] Started GET /queue/dd2da7824e5b7a329f746647e71d126f >[negroni] Completed 303 See Other in 125.363µs >[negroni] Started GET /blockvolumes/15731bca9a658eaf568941f44f58894b >[negroni] Completed 200 OK in 3.157475ms >[kubeexec] DEBUG 2018/08/19 13:35:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:35:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 200 OK in 149.621µs >[kubeexec] DEBUG 2018/08/19 13:35:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 32min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:35:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 39min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:35:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 200 OK in 118.744µs >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 200 OK in 125.407µs >[kubeexec] DEBUG 2018/08/19 13:35:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_7fc36e74a20895372dd7557f554cacd7 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:42e5add2-d6e9-4bf7-b2a4-e3d1a97b63cc", "USERNAME": "42e5add2-d6e9-4bf7-b2a4-e3d1a97b63cc", "PASSWORD": "ad672ed2-7f52-4a6f-a4f8-a63d8e294aaf", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:35:23 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:35:23 asynchttp.go:292: Completed job b7137a316645ccb9671431352495c604 in 4.96891584s >[negroni] Started GET /queue/b7137a316645ccb9671431352495c604 >[negroni] Completed 303 See Other in 125.789µs >[negroni] Started GET /blockvolumes/7fc36e74a20895372dd7557f554cacd7 >[negroni] Completed 200 OK in 3.529006ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.420251ms >[asynchttp] INFO 2018/08/19 13:35:47 asynchttp.go:288: Started job 98521e7b52763a79c0ea6baf0ac21a7c >[heketi] INFO 2018/08/19 13:35:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/98521e7b52763a79c0ea6baf0ac21a7c >[negroni] Completed 200 OK in 110.13µs >[cmdexec] INFO 2018/08/19 13:35:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 33min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:35:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 39min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:35:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:35:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/98521e7b52763a79c0ea6baf0ac21a7c >[negroni] Completed 200 OK in 125.564µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.203093ms >[asynchttp] INFO 2018/08/19 13:35:48 asynchttp.go:288: Started job 6bb2e68bb5ab009cf3891f844acaf45b >[heketi] INFO 2018/08/19 13:35:48 Started async operation: Create Block Volume >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 200 OK in 139.623µs >[cmdexec] INFO 2018/08/19 13:35:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 39min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:35:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:35:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 33min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:35:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/98521e7b52763a79c0ea6baf0ac21a7c >[negroni] Completed 200 OK in 170.388µs >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 200 OK in 119.961µs >[negroni] Started GET /queue/98521e7b52763a79c0ea6baf0ac21a7c >[negroni] Completed 200 OK in 121.946µs >[kubeexec] DEBUG 2018/08/19 13:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_c95ff2b5fb92c57b6e4e5d268f7878f0 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2704aa40-b01d-4230-b33b-011c2d2d0b99", "USERNAME": "2704aa40-b01d-4230-b33b-011c2d2d0b99", "PASSWORD": "002736ad-d5de-4f6f-a59f-fee2df9be215", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:35:50 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:35:50 asynchttp.go:292: Completed job 98521e7b52763a79c0ea6baf0ac21a7c in 3.018930057s >[kubeexec] DEBUG 2018/08/19 13:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:35:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 200 OK in 88.132µs >[kubeexec] DEBUG 2018/08/19 13:35:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 39min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/98521e7b52763a79c0ea6baf0ac21a7c >[negroni] Completed 303 See Other in 137.958µs >[negroni] Started GET /blockvolumes/c95ff2b5fb92c57b6e4e5d268f7878f0 >[negroni] Completed 200 OK in 1.622722ms >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 200 OK in 133.09µs >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 200 OK in 123.48µs >[kubeexec] DEBUG 2018/08/19 13:35:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_12e40f4279f9107e2c86b2c996753885 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7b175260-9240-41df-ac3a-9b5e9c69b9d1", "USERNAME": "7b175260-9240-41df-ac3a-9b5e9c69b9d1", "PASSWORD": "709c4e7e-0e09-4cd0-ad70-8c873d07c51e", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:35:53 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:35:53 asynchttp.go:292: Completed job 6bb2e68bb5ab009cf3891f844acaf45b in 4.7008355s >[negroni] Started GET /queue/6bb2e68bb5ab009cf3891f844acaf45b >[negroni] Completed 303 See Other in 172.399µs >[negroni] Started GET /blockvolumes/12e40f4279f9107e2c86b2c996753885 >[negroni] Completed 200 OK in 2.540362ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 119.096381ms >[asynchttp] INFO 2018/08/19 13:36:17 asynchttp.go:288: Started job cd6f5921bc186cfa6a2ffb32093a5e5b >[heketi] INFO 2018/08/19 13:36:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/cd6f5921bc186cfa6a2ffb32093a5e5b >[negroni] Completed 200 OK in 124.008µs >[cmdexec] INFO 2018/08/19 13:36:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:36:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 39min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:36:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 33min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:36:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cd6f5921bc186cfa6a2ffb32093a5e5b >[negroni] Completed 200 OK in 136.955µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 31.029489ms >[asynchttp] INFO 2018/08/19 13:36:18 asynchttp.go:288: Started job c6f52bad15ec2d8f37bbfc35c81f9599 >[heketi] INFO 2018/08/19 13:36:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 131.285µs >[cmdexec] INFO 2018/08/19 13:36:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 33min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:36:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:36:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/cd6f5921bc186cfa6a2ffb32093a5e5b >[negroni] Completed 200 OK in 129.918µs >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 156.458µs >[negroni] Started GET /queue/cd6f5921bc186cfa6a2ffb32093a5e5b >[negroni] Completed 200 OK in 178.674µs >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 144.146µs >[kubeexec] DEBUG 2018/08/19 13:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_7844d0392a2515ae06a3e51e949527d6 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4c97be9c-dee6-4c74-9040-444eb6a8970f", "USERNAME": "4c97be9c-dee6-4c74-9040-444eb6a8970f", "PASSWORD": "2bac53f6-6683-4fde-bbcb-f84b6dc67bc4", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:36:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:36:21 asynchttp.go:292: Completed job cd6f5921bc186cfa6a2ffb32093a5e5b in 3.251357133s >[kubeexec] DEBUG 2018/08/19 13:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:36:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cd6f5921bc186cfa6a2ffb32093a5e5b >[negroni] Completed 303 See Other in 176.783µs >[negroni] Started GET /blockvolumes/7844d0392a2515ae06a3e51e949527d6 >[negroni] Completed 200 OK in 1.620457ms >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 201.558µs >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 168.705µs >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 200 OK in 124.785µs >[kubeexec] DEBUG 2018/08/19 13:36:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_b8d5592e38246d1aae6102b9e7ca730b ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:86fa3512-9133-44cd-b843-12774f307c5f", "USERNAME": "86fa3512-9133-44cd-b843-12774f307c5f", "PASSWORD": "c72960b1-624f-45ce-b72b-01214aa2f3c9", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:36:24 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:36:24 asynchttp.go:292: Completed job c6f52bad15ec2d8f37bbfc35c81f9599 in 5.09326618s >[negroni] Started GET /queue/c6f52bad15ec2d8f37bbfc35c81f9599 >[negroni] Completed 303 See Other in 134.621µs >[negroni] Started GET /blockvolumes/b8d5592e38246d1aae6102b9e7ca730b >[negroni] Completed 200 OK in 2.551461ms >[heketi] INFO 2018/08/19 13:36:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:36:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:36:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:36:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:36:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:36:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 34min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:36:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:36:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 35.334937ms >[asynchttp] INFO 2018/08/19 13:36:48 asynchttp.go:288: Started job 74170ddf3ff40285b77c986c0d2272bb >[heketi] INFO 2018/08/19 13:36:48 Started async operation: Create Block Volume >[negroni] Started GET /queue/74170ddf3ff40285b77c986c0d2272bb >[negroni] Completed 200 OK in 92.039µs >[cmdexec] INFO 2018/08/19 13:36:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 34min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:36:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:36:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:36:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/74170ddf3ff40285b77c986c0d2272bb >[negroni] Completed 200 OK in 122.065µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 44.370081ms >[asynchttp] INFO 2018/08/19 13:36:49 asynchttp.go:288: Started job 5f5b31f49fc47114ee8a478d58f06141 >[heketi] INFO 2018/08/19 13:36:49 Started async operation: Create Block Volume >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 200 OK in 122.542µs >[cmdexec] INFO 2018/08/19 13:36:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 34min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:36:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 40min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:36:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/74170ddf3ff40285b77c986c0d2272bb >[negroni] Completed 200 OK in 124.522µs >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 200 OK in 136.814µs >[negroni] Started GET /queue/74170ddf3ff40285b77c986c0d2272bb >[negroni] Completed 200 OK in 171.137µs >[kubeexec] DEBUG 2018/08/19 13:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6a59a1e25255430c26c82ddc57cab78f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:83e517dc-7b4d-454c-a51f-a9d4bce7c21a", "USERNAME": "83e517dc-7b4d-454c-a51f-a9d4bce7c21a", "PASSWORD": "4c834d87-ab55-41c7-8c1c-5160e18380da", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:36:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:36:51 asynchttp.go:292: Completed job 74170ddf3ff40285b77c986c0d2272bb in 3.096490927s >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 200 OK in 131.941µs >[kubeexec] DEBUG 2018/08/19 13:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:36:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:36:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 40min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/74170ddf3ff40285b77c986c0d2272bb >[negroni] Completed 303 See Other in 155.227µs >[negroni] Started GET /blockvolumes/6a59a1e25255430c26c82ddc57cab78f >[negroni] Completed 200 OK in 1.375531ms >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 200 OK in 139.263µs >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 200 OK in 139.409µs >[kubeexec] DEBUG 2018/08/19 13:36:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_16fd5990e5fe3065d934b05ba71ba80a ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3469b659-3fe0-46d2-a350-47cd4e3dd8e6", "USERNAME": "3469b659-3fe0-46d2-a350-47cd4e3dd8e6", "PASSWORD": "03f3123e-49ab-44bf-baa6-bfc5586b0238", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:36:54 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:36:54 asynchttp.go:292: Completed job 5f5b31f49fc47114ee8a478d58f06141 in 4.672885922s >[negroni] Started GET /queue/5f5b31f49fc47114ee8a478d58f06141 >[negroni] Completed 303 See Other in 128.557µs >[negroni] Started GET /blockvolumes/16fd5990e5fe3065d934b05ba71ba80a >[negroni] Completed 200 OK in 2.407048ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 28.461907ms >[asynchttp] INFO 2018/08/19 13:37:18 asynchttp.go:288: Started job 5dbb60e5fcbaa4bdb151f3a6d43d984f >[heketi] INFO 2018/08/19 13:37:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/5dbb60e5fcbaa4bdb151f3a6d43d984f >[negroni] Completed 200 OK in 96.119µs >[cmdexec] INFO 2018/08/19 13:37:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 34min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:37:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:37:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:37:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5dbb60e5fcbaa4bdb151f3a6d43d984f >[negroni] Completed 200 OK in 182.585µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 29.290635ms >[asynchttp] INFO 2018/08/19 13:37:19 asynchttp.go:288: Started job b138b89c5daa8c9b1f4af84e70fa228c >[heketi] INFO 2018/08/19 13:37:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 200 OK in 140.903µs >[cmdexec] INFO 2018/08/19 13:37:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 34min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:37:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:37:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/5dbb60e5fcbaa4bdb151f3a6d43d984f >[negroni] Completed 200 OK in 139.798µs >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 200 OK in 139.078µs >[negroni] Started GET /queue/5dbb60e5fcbaa4bdb151f3a6d43d984f >[negroni] Completed 200 OK in 141.126µs >[kubeexec] DEBUG 2018/08/19 13:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_da056435679787df0bf2bda1e204c05b ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a9e034b4-58f1-4103-a296-21b115a441fc", "USERNAME": "a9e034b4-58f1-4103-a296-21b115a441fc", "PASSWORD": "a2acc902-6e45-4d80-af39-ff294037d538", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 200 OK in 124.701µs >[heketi] INFO 2018/08/19 13:37:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:37:21 asynchttp.go:292: Completed job 5dbb60e5fcbaa4bdb151f3a6d43d984f in 3.194605375s >[kubeexec] DEBUG 2018/08/19 13:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:37:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5dbb60e5fcbaa4bdb151f3a6d43d984f >[negroni] Completed 303 See Other in 127.193µs >[negroni] Started GET /blockvolumes/da056435679787df0bf2bda1e204c05b >[negroni] Completed 200 OK in 1.528806ms >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 200 OK in 131.948µs >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 200 OK in 210.592µs >[kubeexec] DEBUG 2018/08/19 13:37:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_df45af3184b81d81a44c208ac3c227ed ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7b1f5e61-96d2-4dae-aac0-3511f1815ec7", "USERNAME": "7b1f5e61-96d2-4dae-aac0-3511f1815ec7", "PASSWORD": "d3fbb496-845b-4063-a5fd-25d7a16b4dcb", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:37:24 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:37:24 asynchttp.go:292: Completed job b138b89c5daa8c9b1f4af84e70fa228c in 4.900220421s >[negroni] Started GET /queue/b138b89c5daa8c9b1f4af84e70fa228c >[negroni] Completed 303 See Other in 182.105µs >[negroni] Started GET /blockvolumes/df45af3184b81d81a44c208ac3c227ed >[negroni] Completed 200 OK in 2.524042ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.357659ms >[asynchttp] INFO 2018/08/19 13:37:48 asynchttp.go:288: Started job 8e7eeeff75aabaf19b4ddbcc43238655 >[heketi] INFO 2018/08/19 13:37:48 Started async operation: Create Block Volume >[negroni] Started GET /queue/8e7eeeff75aabaf19b4ddbcc43238655 >[negroni] Completed 200 OK in 139.527µs >[cmdexec] INFO 2018/08/19 13:37:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:37:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 35min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:37:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:37:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8e7eeeff75aabaf19b4ddbcc43238655 >[negroni] Completed 200 OK in 131.317µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.32734ms >[asynchttp] INFO 2018/08/19 13:37:50 asynchttp.go:288: Started job 85cee28eb276a970bb8905d4a1797b17 >[heketi] INFO 2018/08/19 13:37:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 102.371µs >[cmdexec] INFO 2018/08/19 13:37:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8e7eeeff75aabaf19b4ddbcc43238655 >[negroni] Completed 200 OK in 122.806µs >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 144.157µs >[negroni] Started GET /queue/8e7eeeff75aabaf19b4ddbcc43238655 >[negroni] Completed 200 OK in 159.178µs >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 113.453µs >[kubeexec] DEBUG 2018/08/19 13:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_17c3d6104afa4ed3059ba18208bc3505 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:759860ad-6512-4945-9929-9be5e9e171e3", "USERNAME": "759860ad-6512-4945-9929-9be5e9e171e3", "PASSWORD": "7be1d2ad-cf88-43eb-8ddd-f33271c1e469", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:37:52 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:37:52 asynchttp.go:292: Completed job 8e7eeeff75aabaf19b4ddbcc43238655 in 3.319410416s >[kubeexec] DEBUG 2018/08/19 13:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:37:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 41min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:37:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 35min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:37:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:37:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 41min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8e7eeeff75aabaf19b4ddbcc43238655 >[negroni] Completed 303 See Other in 202.688µs >[negroni] Started GET /blockvolumes/17c3d6104afa4ed3059ba18208bc3505 >[negroni] Completed 200 OK in 1.608106ms >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 119.863µs >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 141.584µs >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 200 OK in 177.371µs >[kubeexec] DEBUG 2018/08/19 13:37:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_362f05b1f651aa71145199154e0eaa5d ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7963ee4e-dc60-4220-a4d9-04c0eafe62ee", "USERNAME": "7963ee4e-dc60-4220-a4d9-04c0eafe62ee", "PASSWORD": "1797f8f5-be90-4133-ba30-5a1021abba65", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:37:55 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:37:55 asynchttp.go:292: Completed job 85cee28eb276a970bb8905d4a1797b17 in 5.619714181s >[negroni] Started GET /queue/85cee28eb276a970bb8905d4a1797b17 >[negroni] Completed 303 See Other in 129.185µs >[negroni] Started GET /blockvolumes/362f05b1f651aa71145199154e0eaa5d >[negroni] Completed 200 OK in 2.899807ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 31.710408ms >[asynchttp] INFO 2018/08/19 13:38:19 asynchttp.go:288: Started job 34110899309d602786158ae265eaf235 >[heketi] INFO 2018/08/19 13:38:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/34110899309d602786158ae265eaf235 >[negroni] Completed 200 OK in 120.152µs >[cmdexec] INFO 2018/08/19 13:38:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:38:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 35min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:38:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:38:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/34110899309d602786158ae265eaf235 >[negroni] Completed 200 OK in 148.502µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 43.298851ms >[asynchttp] INFO 2018/08/19 13:38:20 asynchttp.go:288: Started job 66ff40eb0437d18abb2e003d8328b357 >[heketi] INFO 2018/08/19 13:38:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 168.324µs >[cmdexec] INFO 2018/08/19 13:38:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:38:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/34110899309d602786158ae265eaf235 >[negroni] Completed 200 OK in 148.418µs >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 161.789µs >[negroni] Started GET /queue/34110899309d602786158ae265eaf235 >[negroni] Completed 200 OK in 148.51µs >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 248.432µs >[kubeexec] DEBUG 2018/08/19 13:38:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_5aea9e0a056ac3eb932a23280155e6e0 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e9c29736-de62-47b3-8e8a-6b425ef59196", "USERNAME": "e9c29736-de62-47b3-8e8a-6b425ef59196", "PASSWORD": "55e35271-f1e9-495f-8ac5-55c653e01e1a", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:38:22 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:38:22 asynchttp.go:292: Completed job 34110899309d602786158ae265eaf235 in 3.555185239s >[kubeexec] DEBUG 2018/08/19 13:38:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:38:22 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 35min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:38:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/34110899309d602786158ae265eaf235 >[negroni] Completed 303 See Other in 202.874µs >[negroni] Started GET /blockvolumes/5aea9e0a056ac3eb932a23280155e6e0 >[negroni] Completed 200 OK in 1.59996ms >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 176.197µs >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 169.718µs >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 200 OK in 133.87µs >[kubeexec] DEBUG 2018/08/19 13:38:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_74fd56a66f2d8c39995c5723c3aa4fb9 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c15170eb-0b66-40d8-894b-04ecea9125b6", "USERNAME": "c15170eb-0b66-40d8-894b-04ecea9125b6", "PASSWORD": "d8470f9d-da15-448d-bfb8-12f3c69fd0ee", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:38:26 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:38:26 asynchttp.go:292: Completed job 66ff40eb0437d18abb2e003d8328b357 in 5.790990124s >[negroni] Started GET /queue/66ff40eb0437d18abb2e003d8328b357 >[negroni] Completed 303 See Other in 145.722µs >[negroni] Started GET /blockvolumes/74fd56a66f2d8c39995c5723c3aa4fb9 >[negroni] Completed 200 OK in 3.207812ms >[heketi] INFO 2018/08/19 13:38:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:38:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:38:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:38:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:38:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:38:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 36min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:38:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:38:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 27.998119ms >[asynchttp] INFO 2018/08/19 13:38:49 asynchttp.go:288: Started job 7e59f6829c4f45bb591d7f23aee5ed2a >[heketi] INFO 2018/08/19 13:38:49 Started async operation: Create Block Volume >[negroni] Started GET /queue/7e59f6829c4f45bb591d7f23aee5ed2a >[negroni] Completed 200 OK in 167.584µs >[cmdexec] INFO 2018/08/19 13:38:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 36min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:38:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:38:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:38:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7e59f6829c4f45bb591d7f23aee5ed2a >[negroni] Completed 200 OK in 120.088µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.927252ms >[asynchttp] INFO 2018/08/19 13:38:50 asynchttp.go:288: Started job 17a55da67e441441ebbbcc2ce5153b76 >[heketi] INFO 2018/08/19 13:38:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 134.922µs >[cmdexec] INFO 2018/08/19 13:38:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7e59f6829c4f45bb591d7f23aee5ed2a >[negroni] Completed 200 OK in 144.556µs >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 114.149µs >[negroni] Started GET /queue/7e59f6829c4f45bb591d7f23aee5ed2a >[negroni] Completed 200 OK in 137.26µs >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 117.644µs >[kubeexec] DEBUG 2018/08/19 13:38:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_2ed85b05f740a252aec1aedc9c9cc39f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3c3d17aa-37d0-482d-a47b-e9c99b69cd88", "USERNAME": "3c3d17aa-37d0-482d-a47b-e9c99b69cd88", "PASSWORD": "7129da9b-b69b-4dfe-88a7-00be4dde9854", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:38:53 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:38:53 asynchttp.go:292: Completed job 7e59f6829c4f45bb591d7f23aee5ed2a in 3.569550576s >[kubeexec] DEBUG 2018/08/19 13:38:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:38:53 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 42min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:38:53 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:38:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 36min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:38:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7e59f6829c4f45bb591d7f23aee5ed2a >[negroni] Completed 303 See Other in 139.195µs >[negroni] Started GET /blockvolumes/2ed85b05f740a252aec1aedc9c9cc39f >[negroni] Completed 200 OK in 1.609519ms >[kubeexec] DEBUG 2018/08/19 13:38:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 42min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 136.608µs >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 140.776µs >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 200 OK in 166.162µs >[kubeexec] DEBUG 2018/08/19 13:38:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_38144b016066102d537b40d29e9343b1 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c828ccdf-a025-43c9-b5bb-ae7a79e5a0de", "USERNAME": "c828ccdf-a025-43c9-b5bb-ae7a79e5a0de", "PASSWORD": "7c4b49e4-98f9-46c7-9f7b-c8636ca5004e", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:38:56 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:38:56 asynchttp.go:292: Completed job 17a55da67e441441ebbbcc2ce5153b76 in 5.808826161s >[negroni] Started GET /queue/17a55da67e441441ebbbcc2ce5153b76 >[negroni] Completed 303 See Other in 199.714µs >[negroni] Started GET /blockvolumes/38144b016066102d537b40d29e9343b1 >[negroni] Completed 200 OK in 2.560498ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.38261ms >[asynchttp] INFO 2018/08/19 13:39:20 asynchttp.go:288: Started job a0a8a89d5584907a23f3f249b8c536f2 >[heketi] INFO 2018/08/19 13:39:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/a0a8a89d5584907a23f3f249b8c536f2 >[negroni] Completed 200 OK in 116.814µs >[cmdexec] INFO 2018/08/19 13:39:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:39:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 36min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:39:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:39:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a0a8a89d5584907a23f3f249b8c536f2 >[negroni] Completed 200 OK in 123.453µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 37.431851ms >[asynchttp] INFO 2018/08/19 13:39:21 asynchttp.go:288: Started job 57163f397b819c37ede1bb206a02a1dd >[heketi] INFO 2018/08/19 13:39:21 Started async operation: Create Block Volume >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 120.29µs >[cmdexec] INFO 2018/08/19 13:39:21 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:39:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 36min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:39:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a0a8a89d5584907a23f3f249b8c536f2 >[negroni] Completed 200 OK in 118.794µs >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 136.795µs >[negroni] Started GET /queue/a0a8a89d5584907a23f3f249b8c536f2 >[negroni] Completed 200 OK in 117.425µs >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 123.452µs >[kubeexec] DEBUG 2018/08/19 13:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_ac3ce1442fa0c85cbbcecedb2bc7b036 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3a2cbcb7-7a00-4c64-b048-78dc92e222b4", "USERNAME": "3a2cbcb7-7a00-4c64-b048-78dc92e222b4", "PASSWORD": "790c301d-8e96-4a87-8745-e4a29fd5228c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:39:23 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:39:23 asynchttp.go:292: Completed job a0a8a89d5584907a23f3f249b8c536f2 in 3.589328064s >[kubeexec] DEBUG 2018/08/19 13:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:39:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a0a8a89d5584907a23f3f249b8c536f2 >[negroni] Completed 303 See Other in 136.864µs >[negroni] Started GET /blockvolumes/ac3ce1442fa0c85cbbcecedb2bc7b036 >[negroni] Completed 200 OK in 1.368205ms >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 159.011µs >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 118.968µs >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 200 OK in 124.752µs >[kubeexec] DEBUG 2018/08/19 13:39:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_1df8bad146cdce8f9e0eddc23e41a146 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bf6d1b36-6074-4f4a-9afe-24c375650cb4", "USERNAME": "bf6d1b36-6074-4f4a-9afe-24c375650cb4", "PASSWORD": "68f01fc8-06ad-43d4-aa4d-bf87bea3ad23", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:39:26 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:39:26 asynchttp.go:292: Completed job 57163f397b819c37ede1bb206a02a1dd in 5.670202307s >[negroni] Started GET /queue/57163f397b819c37ede1bb206a02a1dd >[negroni] Completed 303 See Other in 126.087µs >[negroni] Started GET /blockvolumes/1df8bad146cdce8f9e0eddc23e41a146 >[negroni] Completed 200 OK in 2.394169ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 73.987784ms >[asynchttp] INFO 2018/08/19 13:39:50 asynchttp.go:288: Started job 170e35cdc292f2ac12a24b42975ec138 >[heketi] INFO 2018/08/19 13:39:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/170e35cdc292f2ac12a24b42975ec138 >[negroni] Completed 200 OK in 105.258µs >[cmdexec] INFO 2018/08/19 13:39:50 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 37min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:39:50 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:39:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:39:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/170e35cdc292f2ac12a24b42975ec138 >[negroni] Completed 200 OK in 126.712µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 89.074146ms >[asynchttp] INFO 2018/08/19 13:39:51 asynchttp.go:288: Started job ab52e9eb020dd4c124018d0092740c65 >[heketi] INFO 2018/08/19 13:39:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 110.7µs >[cmdexec] INFO 2018/08/19 13:39:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 43min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:39:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:39:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 37min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:39:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/170e35cdc292f2ac12a24b42975ec138 >[negroni] Completed 200 OK in 164.442µs >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 225.27µs >[negroni] Started GET /queue/170e35cdc292f2ac12a24b42975ec138 >[negroni] Completed 200 OK in 123.303µs >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 129.933µs >[kubeexec] DEBUG 2018/08/19 13:39:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_422c8abe0536b2d450fd0dc57785b869 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f99d52f1-a081-4416-910b-9fc3bb523537", "USERNAME": "f99d52f1-a081-4416-910b-9fc3bb523537", "PASSWORD": "5659bdfb-7c19-44c1-a4ec-c3fa47510f29", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 13:39:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:39:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 13:39:54 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:39:54 asynchttp.go:292: Completed job 170e35cdc292f2ac12a24b42975ec138 in 3.808081975s >[kubeexec] DEBUG 2018/08/19 13:39:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 43min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/170e35cdc292f2ac12a24b42975ec138 >[negroni] Completed 303 See Other in 193.95µs >[negroni] Started GET /blockvolumes/422c8abe0536b2d450fd0dc57785b869 >[negroni] Completed 200 OK in 3.228155ms >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 171.492µs >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 116.873µs >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 200 OK in 145.438µs >[kubeexec] DEBUG 2018/08/19 13:39:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6857ffa4f448679153fb0070c5462ceb ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:81237559-594b-47de-8176-bcd7d9de539f", "USERNAME": "81237559-594b-47de-8176-bcd7d9de539f", "PASSWORD": "0ba08e74-dafd-44a2-93b3-b4aa14afd4df", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:39:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:39:57 asynchttp.go:292: Completed job ab52e9eb020dd4c124018d0092740c65 in 5.876388s >[negroni] Started GET /queue/ab52e9eb020dd4c124018d0092740c65 >[negroni] Completed 303 See Other in 191.237µs >[negroni] Started GET /blockvolumes/6857ffa4f448679153fb0070c5462ceb >[negroni] Completed 200 OK in 2.138008ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 30.36487ms >[asynchttp] INFO 2018/08/19 13:40:20 asynchttp.go:288: Started job 7bcac1190de344e5e156ce1c945bf5d1 >[heketi] INFO 2018/08/19 13:40:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 200 OK in 100.451µs >[cmdexec] INFO 2018/08/19 13:40:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:40:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:40:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 37min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:40:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 200 OK in 143.842µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 27.681476ms >[asynchttp] INFO 2018/08/19 13:40:21 asynchttp.go:288: Started job 66f54fce728c963773a33a9a170d918d >[heketi] INFO 2018/08/19 13:40:21 Started async operation: Create Block Volume >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 102.078µs >[cmdexec] INFO 2018/08/19 13:40:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 200 OK in 156.606µs >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 126.329µs >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 200 OK in 121.539µs >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 212.097µs >[kubeexec] DEBUG 2018/08/19 13:40:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_9f01918e213b0656075e1507a30d5f62 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:5f7281e2-e277-4139-b647-f24d40f2de7e", "USERNAME": "5f7281e2-e277-4139-b647-f24d40f2de7e", "PASSWORD": "5d1752b1-8de9-4784-afc9-88a927df6ec1", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 13:40:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:40:24 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:40:24 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 37min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:40:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 200 OK in 93.522µs >[heketi] INFO 2018/08/19 13:40:24 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:40:24 asynchttp.go:292: Completed job 7bcac1190de344e5e156ce1c945bf5d1 in 4.015659875s >[kubeexec] DEBUG 2018/08/19 13:40:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 137.042µs >[negroni] Started GET /queue/7bcac1190de344e5e156ce1c945bf5d1 >[negroni] Completed 303 See Other in 203.604µs >[negroni] Started GET /blockvolumes/9f01918e213b0656075e1507a30d5f62 >[negroni] Completed 200 OK in 2.652933ms >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 121.9µs >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 125.184µs >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 200 OK in 149.863µs >[kubeexec] DEBUG 2018/08/19 13:40:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_74d2893062db7720689e628108dfccb3 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2e4b62cd-3049-42bc-9ba2-a3f4c30bf1a2", "USERNAME": "2e4b62cd-3049-42bc-9ba2-a3f4c30bf1a2", "PASSWORD": "bff9b25d-d212-495a-8dbc-0b94c9140b6c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:40:27 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:40:27 asynchttp.go:292: Completed job 66f54fce728c963773a33a9a170d918d in 6.046253403s >[negroni] Started GET /queue/66f54fce728c963773a33a9a170d918d >[negroni] Completed 303 See Other in 126.155µs >[negroni] Started GET /blockvolumes/74d2893062db7720689e628108dfccb3 >[negroni] Completed 200 OK in 2.892763ms >[heketi] INFO 2018/08/19 13:40:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:40:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:40:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:40:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:40:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:40:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 38min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:40:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:40:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 34.088777ms >[asynchttp] INFO 2018/08/19 13:40:51 asynchttp.go:288: Started job e639a1fe23121ad906c2c6d3048cd834 >[heketi] INFO 2018/08/19 13:40:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 200 OK in 140.928µs >[cmdexec] INFO 2018/08/19 13:40:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:40:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 38min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:40:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:40:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 200 OK in 154.732µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.319098ms >[asynchttp] INFO 2018/08/19 13:40:52 asynchttp.go:288: Started job 75f2617401a4d0df0d028c51e91447ba >[heketi] INFO 2018/08/19 13:40:52 Started async operation: Create Block Volume >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 130.43µs >[cmdexec] INFO 2018/08/19 13:40:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 38min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:40:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 44min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:40:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 200 OK in 138.837µs >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 149.987µs >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 200 OK in 117.018µs >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 165.782µs >[kubeexec] DEBUG 2018/08/19 13:40:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_2d0316327d767f058260ad40f3714ffb ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d6f78b26-6f41-4ced-836e-a01c16edd8c1", "USERNAME": "d6f78b26-6f41-4ced-836e-a01c16edd8c1", "PASSWORD": "af850434-d6b4-446c-a13d-bae8a02005ad", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 13:40:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:40:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:40:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 44min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 200 OK in 97.432µs >[heketi] INFO 2018/08/19 13:40:55 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:40:55 asynchttp.go:292: Completed job e639a1fe23121ad906c2c6d3048cd834 in 4.092568275s >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 237.211µs >[negroni] Started GET /queue/e639a1fe23121ad906c2c6d3048cd834 >[negroni] Completed 303 See Other in 180.345µs >[negroni] Started GET /blockvolumes/2d0316327d767f058260ad40f3714ffb >[negroni] Completed 200 OK in 2.623406ms >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 121.924µs >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 173.495µs >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 200 OK in 136.762µs >[kubeexec] DEBUG 2018/08/19 13:40:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_17adc142cc5ae482f4c0fc1db6223ec3 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2fad0445-da6e-473e-a53a-cacf1ac1f690", "USERNAME": "2fad0445-da6e-473e-a53a-cacf1ac1f690", "PASSWORD": "9be4962b-3f5a-4c6d-8637-991b3cb93c44", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:40:58 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:40:58 asynchttp.go:292: Completed job 75f2617401a4d0df0d028c51e91447ba in 6.212422665s >[negroni] Started GET /queue/75f2617401a4d0df0d028c51e91447ba >[negroni] Completed 303 See Other in 135.865µs >[negroni] Started GET /blockvolumes/17adc142cc5ae482f4c0fc1db6223ec3 >[negroni] Completed 200 OK in 2.61603ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 40.143451ms >[asynchttp] INFO 2018/08/19 13:41:21 asynchttp.go:288: Started job faa42876fe1d102369c8723831bb98c4 >[heketi] INFO 2018/08/19 13:41:21 Started async operation: Create Block Volume >[negroni] Started GET /queue/faa42876fe1d102369c8723831bb98c4 >[negroni] Completed 200 OK in 126.532µs >[cmdexec] INFO 2018/08/19 13:41:21 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 45min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:41:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:41:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 38min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:41:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/faa42876fe1d102369c8723831bb98c4 >[negroni] Completed 200 OK in 184.355µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 32.04124ms >[asynchttp] INFO 2018/08/19 13:41:22 asynchttp.go:288: Started job 8fdf464ff46cadbadbec85f2a7489743 >[heketi] INFO 2018/08/19 13:41:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 91.149µs >[cmdexec] INFO 2018/08/19 13:41:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/faa42876fe1d102369c8723831bb98c4 >[negroni] Completed 200 OK in 143.363µs >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 118.901µs >[negroni] Started GET /queue/faa42876fe1d102369c8723831bb98c4 >[negroni] Completed 200 OK in 115.437µs >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 119.809µs >[kubeexec] DEBUG 2018/08/19 13:41:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_f82527f1688e27911d0830166455da35 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c0aaf20e-d864-46cc-95d6-610406e7ade3", "USERNAME": "c0aaf20e-d864-46cc-95d6-610406e7ade3", "PASSWORD": "a6a7776c-7e5e-471e-9ba7-12a0e270790a", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:41:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:41:25 asynchttp.go:292: Completed job faa42876fe1d102369c8723831bb98c4 in 3.880090868s >[kubeexec] DEBUG 2018/08/19 13:41:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/faa42876fe1d102369c8723831bb98c4 >[negroni] Completed 303 See Other in 106.911µs >[negroni] Started GET /blockvolumes/f82527f1688e27911d0830166455da35 >[cmdexec] INFO 2018/08/19 13:41:25 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Completed 200 OK in 1.843265ms >[kubeexec] DEBUG 2018/08/19 13:41:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 38min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:41:25 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 149.428µs >[kubeexec] DEBUG 2018/08/19 13:41:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 45min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:41:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 136.353µs >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 141.174µs >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 200 OK in 158.568µs >[kubeexec] DEBUG 2018/08/19 13:41:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_d729b77a66055888a2f3b032b41a71da ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d1d7da31-c0a7-406d-a011-609503c8bae5", "USERNAME": "d1d7da31-c0a7-406d-a011-609503c8bae5", "PASSWORD": "1d5ff2df-1651-4581-811e-36f854e8aaaf", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:41:29 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:41:29 asynchttp.go:292: Completed job 8fdf464ff46cadbadbec85f2a7489743 in 6.826891615s >[negroni] Started GET /queue/8fdf464ff46cadbadbec85f2a7489743 >[negroni] Completed 303 See Other in 158.485µs >[negroni] Started GET /blockvolumes/d729b77a66055888a2f3b032b41a71da >[negroni] Completed 200 OK in 2.485293ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 28.702683ms >[asynchttp] INFO 2018/08/19 13:41:51 asynchttp.go:288: Started job 65730c59a5c9b70677f4d26bbbde4033 >[heketi] INFO 2018/08/19 13:41:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/65730c59a5c9b70677f4d26bbbde4033 >[negroni] Completed 200 OK in 95.932µs >[cmdexec] INFO 2018/08/19 13:41:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 39min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:41:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:41:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 45min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:41:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/65730c59a5c9b70677f4d26bbbde4033 >[negroni] Completed 200 OK in 187.534µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.073893ms >[asynchttp] INFO 2018/08/19 13:41:52 asynchttp.go:288: Started job f0b58ee3ac86a62f2b514a8bf67c544d >[heketi] INFO 2018/08/19 13:41:52 Started async operation: Create Block Volume >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 123.477µs >[cmdexec] INFO 2018/08/19 13:41:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 45min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:41:53 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:41:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 39min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:41:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/65730c59a5c9b70677f4d26bbbde4033 >[negroni] Completed 200 OK in 159.478µs >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 117.545µs >[negroni] Started GET /queue/65730c59a5c9b70677f4d26bbbde4033 >[negroni] Completed 200 OK in 143.733µs >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 119.529µs >[kubeexec] DEBUG 2018/08/19 13:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_412fb2fd52501f79c87118cdfb5c0989 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7e6bc6d7-75e6-4ed3-8d7c-492f5766dfd7", "USERNAME": "7e6bc6d7-75e6-4ed3-8d7c-492f5766dfd7", "PASSWORD": "ce4eecf3-2595-4dcd-9f9b-2970d52a529a", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:41:55 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:41:55 asynchttp.go:292: Completed job 65730c59a5c9b70677f4d26bbbde4033 in 3.877430958s >[kubeexec] DEBUG 2018/08/19 13:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:41:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/65730c59a5c9b70677f4d26bbbde4033 >[negroni] Completed 303 See Other in 100.005µs >[negroni] Started GET /blockvolumes/412fb2fd52501f79c87118cdfb5c0989 >[negroni] Completed 200 OK in 1.288533ms >[kubeexec] DEBUG 2018/08/19 13:41:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 45min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 104.041µs >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 192.068µs >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 145.456µs >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 200 OK in 133.644µs >[kubeexec] DEBUG 2018/08/19 13:41:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_fd03b0949c724da97837aa650cd0aee8 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e2edddb8-81d2-4faf-b4d2-85e1ebdabec3", "USERNAME": "e2edddb8-81d2-4faf-b4d2-85e1ebdabec3", "PASSWORD": "0a92f464-3f71-443f-b1dc-e10c0f3071a8", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:41:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:41:59 asynchttp.go:292: Completed job f0b58ee3ac86a62f2b514a8bf67c544d in 6.349808172s >[negroni] Started GET /queue/f0b58ee3ac86a62f2b514a8bf67c544d >[negroni] Completed 303 See Other in 171.907µs >[negroni] Started GET /blockvolumes/fd03b0949c724da97837aa650cd0aee8 >[negroni] Completed 200 OK in 2.711266ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.074023ms >[asynchttp] INFO 2018/08/19 13:42:22 asynchttp.go:288: Started job 259d476d76ccbf8a50c8d64b7d15165c >[heketi] INFO 2018/08/19 13:42:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 200 OK in 98.373µs >[cmdexec] INFO 2018/08/19 13:42:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:42:22 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 39min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:42:22 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:42:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 200 OK in 129.081µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 33.57343ms >[asynchttp] INFO 2018/08/19 13:42:23 asynchttp.go:288: Started job 6f5e538b8cd50a4bff211d2c6ab892ff >[heketi] INFO 2018/08/19 13:42:23 Started async operation: Create Block Volume >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 141.457µs >[cmdexec] INFO 2018/08/19 13:42:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:42:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 39min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:42:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 200 OK in 140.094µs >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 143.245µs >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 200 OK in 122.785µs >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 122.473µs >[kubeexec] DEBUG 2018/08/19 13:42:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_11d5b7c5451d69798b5d221581b6e961 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:38779437-f917-4936-bc39-fedaedf06eed", "USERNAME": "38779437-f917-4936-bc39-fedaedf06eed", "PASSWORD": "6d53c297-bcec-44d6-b965-10f06c1f1983", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 200 OK in 124.632µs >[heketi] INFO 2018/08/19 13:42:26 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:42:26 asynchttp.go:292: Completed job 259d476d76ccbf8a50c8d64b7d15165c in 4.00758166s >[kubeexec] DEBUG 2018/08/19 13:42:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:42:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 117.805µs >[kubeexec] DEBUG 2018/08/19 13:42:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/259d476d76ccbf8a50c8d64b7d15165c >[negroni] Completed 303 See Other in 140.656µs >[negroni] Started GET /blockvolumes/11d5b7c5451d69798b5d221581b6e961 >[negroni] Completed 200 OK in 1.592888ms >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 153.995µs >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 162.292µs >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 200 OK in 131.461µs >[kubeexec] DEBUG 2018/08/19 13:42:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_349505850fae103ea3ee71cc0056af44 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ee180756-38de-4bec-8422-c1ab6b9695e4", "USERNAME": "ee180756-38de-4bec-8422-c1ab6b9695e4", "PASSWORD": "69788eb0-c159-4162-b655-972614824190", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:42:29 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:42:29 asynchttp.go:292: Completed job 6f5e538b8cd50a4bff211d2c6ab892ff in 6.512273868s >[negroni] Started GET /queue/6f5e538b8cd50a4bff211d2c6ab892ff >[negroni] Completed 303 See Other in 217.068µs >[negroni] Started GET /blockvolumes/349505850fae103ea3ee71cc0056af44 >[negroni] Completed 200 OK in 2.455396ms >[heketi] INFO 2018/08/19 13:42:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:42:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:42:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:42:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:42:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:42:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 40min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:42:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:42:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 44.481242ms >[asynchttp] INFO 2018/08/19 13:42:52 asynchttp.go:288: Started job f5915b4dd96e5e48e4f791d9c1120746 >[heketi] INFO 2018/08/19 13:42:52 Started async operation: Create Block Volume >[negroni] Started GET /queue/f5915b4dd96e5e48e4f791d9c1120746 >[negroni] Completed 200 OK in 136.282µs >[cmdexec] INFO 2018/08/19 13:42:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:42:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 40min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:42:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:42:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f5915b4dd96e5e48e4f791d9c1120746 >[negroni] Completed 200 OK in 171.598µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 560.315214ms >[asynchttp] INFO 2018/08/19 13:42:54 asynchttp.go:288: Started job 2ae6f21ff9ee8ad20fa6c1bf39396e7a >[heketi] INFO 2018/08/19 13:42:54 Started async operation: Create Block Volume >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 115.423µs >[cmdexec] INFO 2018/08/19 13:42:54 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 40min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:42:54 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f5915b4dd96e5e48e4f791d9c1120746 >[negroni] Completed 200 OK in 122.209µs >[kubeexec] DEBUG 2018/08/19 13:42:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 46min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:42:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 198.749µs >[negroni] Started GET /queue/f5915b4dd96e5e48e4f791d9c1120746 >[negroni] Completed 200 OK in 121.348µs >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 147.997µs >[kubeexec] DEBUG 2018/08/19 13:42:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6f10c2abd418450d577f0af07627f1a7 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2acabd9e-fe9b-434a-8714-7608ef2ec285", "USERNAME": "2acabd9e-fe9b-434a-8714-7608ef2ec285", "PASSWORD": "b2448f5f-4637-4751-9e34-b1c2e1cac259", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:42:56 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:42:56 asynchttp.go:292: Completed job f5915b4dd96e5e48e4f791d9c1120746 in 3.82828318s >[kubeexec] DEBUG 2018/08/19 13:42:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:42:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f5915b4dd96e5e48e4f791d9c1120746 >[negroni] Completed 303 See Other in 263.695µs >[negroni] Started GET /blockvolumes/6f10c2abd418450d577f0af07627f1a7 >[negroni] Completed 200 OK in 1.57122ms >[kubeexec] DEBUG 2018/08/19 13:42:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 46min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 164.865µs >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 180.011µs >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 200 OK in 130.846µs >[kubeexec] DEBUG 2018/08/19 13:43:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_6649d263eb3b97a2540cae45284cdfc3 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:881695a1-ebd1-499a-8b7a-8b98c430c5aa", "USERNAME": "881695a1-ebd1-499a-8b7a-8b98c430c5aa", "PASSWORD": "49f001b8-8b17-4121-a12b-13ffd7f5d06f", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:43:00 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:43:00 asynchttp.go:292: Completed job 2ae6f21ff9ee8ad20fa6c1bf39396e7a in 5.908188506s >[negroni] Started GET /queue/2ae6f21ff9ee8ad20fa6c1bf39396e7a >[negroni] Completed 303 See Other in 180.385µs >[negroni] Started GET /blockvolumes/6649d263eb3b97a2540cae45284cdfc3 >[negroni] Completed 200 OK in 2.08945ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.930361ms >[asynchttp] INFO 2018/08/19 13:43:22 asynchttp.go:288: Started job 398baf174485fa8b12e5180af4b6b020 >[heketi] INFO 2018/08/19 13:43:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 200 OK in 97.048µs >[cmdexec] INFO 2018/08/19 13:43:22 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 47min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:43:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 40min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:43:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:43:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 200 OK in 180.872µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 33.723415ms >[asynchttp] INFO 2018/08/19 13:43:24 asynchttp.go:288: Started job 193769eb79af963441dde2845ef87cf3 >[heketi] INFO 2018/08/19 13:43:24 Started async operation: Create Block Volume >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 103.587µs >[cmdexec] INFO 2018/08/19 13:43:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 200 OK in 144.896µs >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 131.96µs >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 200 OK in 178.418µs >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 165.484µs >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 200 OK in 150.619µs >[kubeexec] DEBUG 2018/08/19 13:43:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_5232adc74164d25ef827211d61143948 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:21d5a961-7749-4680-8313-3cf8eca53bf7", "USERNAME": "21d5a961-7749-4680-8313-3cf8eca53bf7", "PASSWORD": "3930d4d1-1921-48bd-8834-2d4a235d01ae", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:43:26 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:43:26 asynchttp.go:292: Completed job 398baf174485fa8b12e5180af4b6b020 in 4.070059976s >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 174.135µs >[kubeexec] DEBUG 2018/08/19 13:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:43:27 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 40min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:43:27 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 47min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:43:27 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/398baf174485fa8b12e5180af4b6b020 >[negroni] Completed 303 See Other in 194.514µs >[negroni] Started GET /blockvolumes/5232adc74164d25ef827211d61143948 >[negroni] Completed 200 OK in 1.449049ms >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 145.689µs >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 166.795µs >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 200 OK in 180.271µs >[kubeexec] DEBUG 2018/08/19 13:43:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_8e580ccf0bc89ef18f56f64b8c799381 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b6ad3989-32a5-45c3-bcb2-c3f28f6ffdde", "USERNAME": "b6ad3989-32a5-45c3-bcb2-c3f28f6ffdde", "PASSWORD": "7b1996fe-0919-4f53-88e0-75f63f5bff33", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:43:31 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:43:31 asynchttp.go:292: Completed job 193769eb79af963441dde2845ef87cf3 in 6.992908243s >[negroni] Started GET /queue/193769eb79af963441dde2845ef87cf3 >[negroni] Completed 303 See Other in 171.939µs >[negroni] Started GET /blockvolumes/8e580ccf0bc89ef18f56f64b8c799381 >[negroni] Completed 200 OK in 2.592638ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 41.149785ms >[asynchttp] INFO 2018/08/19 13:43:53 asynchttp.go:288: Started job d9c9f9d5f84619038b81e348e8861fe3 >[heketi] INFO 2018/08/19 13:43:53 Started async operation: Create Block Volume >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 200 OK in 102.074µs >[cmdexec] INFO 2018/08/19 13:43:53 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 47min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:43:53 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 41min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:43:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:43:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 200 OK in 179.205µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 39.286843ms >[asynchttp] INFO 2018/08/19 13:43:54 asynchttp.go:288: Started job 2656b2e29ef795424fcf5422c31d0fdc >[heketi] INFO 2018/08/19 13:43:54 Started async operation: Create Block Volume >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 170.423µs >[cmdexec] INFO 2018/08/19 13:43:54 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 47min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:43:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 200 OK in 166.316µs >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 115.918µs >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 200 OK in 141.903µs >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 156.678µs >[kubeexec] DEBUG 2018/08/19 13:43:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_fa6dd14be87079c3ec8aed51afe284e1 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:abd29c22-8d88-4cd6-aed6-1568d7c73aa8", "USERNAME": "abd29c22-8d88-4cd6-aed6-1568d7c73aa8", "PASSWORD": "f953305b-51f6-4296-943c-9a63b92bef53", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 200 OK in 123.841µs >[heketi] INFO 2018/08/19 13:43:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:43:57 asynchttp.go:292: Completed job d9c9f9d5f84619038b81e348e8861fe3 in 4.015638963s >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 146.558µs >[kubeexec] DEBUG 2018/08/19 13:43:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:43:57 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 41min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:43:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:43:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 47min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/d9c9f9d5f84619038b81e348e8861fe3 >[negroni] Completed 303 See Other in 194.06µs >[negroni] Started GET /blockvolumes/fa6dd14be87079c3ec8aed51afe284e1 >[negroni] Completed 200 OK in 1.448544ms >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 138.546µs >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 121.607µs >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 178.811µs >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 200 OK in 199.413µs >[kubeexec] DEBUG 2018/08/19 13:44:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_5a0ea924ddec99003fd78d507c18bb9f ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c571b38f-2e5f-4a36-ba65-ff9dd7e058f0", "USERNAME": "c571b38f-2e5f-4a36-ba65-ff9dd7e058f0", "PASSWORD": "f9974c4e-5c97-4106-9bcf-7e913c21edee", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:44:01 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:44:01 asynchttp.go:292: Completed job 2656b2e29ef795424fcf5422c31d0fdc in 7.091687536s >[negroni] Started GET /queue/2656b2e29ef795424fcf5422c31d0fdc >[negroni] Completed 303 See Other in 124.418µs >[negroni] Started GET /blockvolumes/5a0ea924ddec99003fd78d507c18bb9f >[negroni] Completed 200 OK in 2.125338ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 122.478722ms >[asynchttp] INFO 2018/08/19 13:44:23 asynchttp.go:288: Started job c783a7c91d0b6609af20ad580013b34a >[heketi] INFO 2018/08/19 13:44:23 Started async operation: Create Block Volume >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 200 OK in 98.331µs >[cmdexec] INFO 2018/08/19 13:44:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 41min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:44:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:44:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:44:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 200 OK in 197.412µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 40.80393ms >[asynchttp] INFO 2018/08/19 13:44:24 asynchttp.go:288: Started job bef82f7b0a2780d830625eba40e67f5f >[heketi] INFO 2018/08/19 13:44:24 Started async operation: Create Block Volume >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 214.964µs >[cmdexec] INFO 2018/08/19 13:44:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 200 OK in 181.549µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 123.034µs >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 200 OK in 155.231µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 167.926µs >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 200 OK in 200.137µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 167.082µs >[kubeexec] DEBUG 2018/08/19 13:44:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_b440488138c3a40af4e004b205227450 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:8f5bfde7-ec14-41a5-ace8-6eed5eafac63", "USERNAME": "8f5bfde7-ec14-41a5-ace8-6eed5eafac63", "PASSWORD": "741d5760-ac04-4084-ba6b-073f0cc695ba", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:44:27 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:44:27 asynchttp.go:292: Completed job c783a7c91d0b6609af20ad580013b34a in 4.251907171s >[kubeexec] DEBUG 2018/08/19 13:44:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:44:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 41min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:44:28 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:44:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c783a7c91d0b6609af20ad580013b34a >[negroni] Completed 303 See Other in 197.682µs >[negroni] Started GET /blockvolumes/b440488138c3a40af4e004b205227450 >[negroni] Completed 200 OK in 2.105158ms >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 124.451µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 130.667µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 174.69µs >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 200 OK in 139.047µs >[kubeexec] DEBUG 2018/08/19 13:44:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_06732d7214751046797deec74963b232 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c7b6e2fd-1b99-4777-aabf-e2b93924ff07", "USERNAME": "c7b6e2fd-1b99-4777-aabf-e2b93924ff07", "PASSWORD": "88cd18ec-92e0-4996-a70a-9eefee0e3200", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:44:32 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:44:32 asynchttp.go:292: Completed job bef82f7b0a2780d830625eba40e67f5f in 7.501635198s >[negroni] Started GET /queue/bef82f7b0a2780d830625eba40e67f5f >[negroni] Completed 303 See Other in 129.146µs >[negroni] Started GET /blockvolumes/06732d7214751046797deec74963b232 >[negroni] Completed 200 OK in 2.229704ms >[heketi] INFO 2018/08/19 13:44:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:44:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:44:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:44:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:44:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:44:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 42min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:44:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:44:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 66.006219ms >[asynchttp] INFO 2018/08/19 13:44:54 asynchttp.go:288: Started job 5b26bb0f8eb05eade780204db934b39b >[heketi] INFO 2018/08/19 13:44:54 Started async operation: Create Block Volume >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 200 OK in 192.959µs >[cmdexec] INFO 2018/08/19 13:44:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:44:54 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:44:54 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 42min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:44:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 200 OK in 144.856µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 41.008734ms >[asynchttp] INFO 2018/08/19 13:44:55 asynchttp.go:288: Started job 32fd34f029206b1a750d053ec6570b6c >[heketi] INFO 2018/08/19 13:44:55 Started async operation: Create Block Volume >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 155.946µs >[cmdexec] INFO 2018/08/19 13:44:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 48min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:44:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 200 OK in 127.116µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 169.378µs >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 200 OK in 115.473µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 193.226µs >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 200 OK in 152.183µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 169.713µs >[kubeexec] DEBUG 2018/08/19 13:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_3c2bf7dddaa88ceb45fed2fe2e9e2244 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a3f47cd3-d84c-45cc-9f60-cf309f439114", "USERNAME": "a3f47cd3-d84c-45cc-9f60-cf309f439114", "PASSWORD": "68f072c9-78c5-46a3-95d8-a83fcb67c145", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:44:58 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:44:58 asynchttp.go:292: Completed job 5b26bb0f8eb05eade780204db934b39b in 4.506747535s >[kubeexec] DEBUG 2018/08/19 13:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:44:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 42min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:44:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:44:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 48min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5b26bb0f8eb05eade780204db934b39b >[negroni] Completed 303 See Other in 131.873µs >[negroni] Started GET /blockvolumes/3c2bf7dddaa88ceb45fed2fe2e9e2244 >[negroni] Completed 200 OK in 1.496345ms >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 135.737µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 156.123µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 159.825µs >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 200 OK in 162.329µs >[kubeexec] DEBUG 2018/08/19 13:45:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_d08597eb999fade0c17e5cae992a8637 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:2dd52a6c-ddec-46ef-a4d6-65a14ca59081", "USERNAME": "2dd52a6c-ddec-46ef-a4d6-65a14ca59081", "PASSWORD": "d041e47f-10b3-4650-ae8d-814d337ba8a9", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:45:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:45:02 asynchttp.go:292: Completed job 32fd34f029206b1a750d053ec6570b6c in 7.652192734s >[negroni] Started GET /queue/32fd34f029206b1a750d053ec6570b6c >[negroni] Completed 303 See Other in 180.469µs >[negroni] Started GET /blockvolumes/d08597eb999fade0c17e5cae992a8637 >[negroni] Completed 200 OK in 2.476306ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 34.981327ms >[asynchttp] INFO 2018/08/19 13:45:24 asynchttp.go:288: Started job 040c1dce4f0f17e9733c272f616a6d14 >[heketi] INFO 2018/08/19 13:45:24 Started async operation: Create Block Volume >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 200 OK in 142.502µs >[cmdexec] INFO 2018/08/19 13:45:24 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 42min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:45:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:45:24 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 49min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:45:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 200 OK in 146.543µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 35.813172ms >[asynchttp] INFO 2018/08/19 13:45:25 asynchttp.go:288: Started job 269f52a7c25f5ff7a9201fd0c976daba >[heketi] INFO 2018/08/19 13:45:25 Started async operation: Create Block Volume >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 121.305µs >[cmdexec] INFO 2018/08/19 13:45:25 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 49min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:45:25 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 42min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:45:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 200 OK in 181.925µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 162.692µs >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 200 OK in 131.684µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 124.401µs >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 200 OK in 209.928µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 131.702µs >[kubeexec] DEBUG 2018/08/19 13:45:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_f157d376514ea9e6f0b0e37f2e5b5612 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ab0fc594-4917-47b5-81a9-0dd6be1bc6b1", "USERNAME": "ab0fc594-4917-47b5-81a9-0dd6be1bc6b1", "PASSWORD": "bcfb1aec-d2ca-458f-b629-c15bcf545835", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:45:28 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:45:28 asynchttp.go:292: Completed job 040c1dce4f0f17e9733c272f616a6d14 in 4.412754671s >[kubeexec] DEBUG 2018/08/19 13:45:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:45:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/040c1dce4f0f17e9733c272f616a6d14 >[negroni] Completed 303 See Other in 127.45µs >[negroni] Started GET /blockvolumes/f157d376514ea9e6f0b0e37f2e5b5612 >[negroni] Completed 200 OK in 1.581233ms >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 139.379µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 183.296µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 152.813µs >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 200 OK in 210.823µs >[kubeexec] DEBUG 2018/08/19 13:45:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_458139b201e6a2548b7a46a0c67dc80b ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bceee9dd-456d-4ecb-9c62-c6dd8366f171", "USERNAME": "bceee9dd-456d-4ecb-9c62-c6dd8366f171", "PASSWORD": "3548a44a-70ad-4886-a02a-cf0ecc5a01ab", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:45:33 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:45:33 asynchttp.go:292: Completed job 269f52a7c25f5ff7a9201fd0c976daba in 7.526541002s >[negroni] Started GET /queue/269f52a7c25f5ff7a9201fd0c976daba >[negroni] Completed 303 See Other in 128.832µs >[negroni] Started GET /blockvolumes/458139b201e6a2548b7a46a0c67dc80b >[negroni] Completed 200 OK in 2.584067ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 132.158468ms >[asynchttp] INFO 2018/08/19 13:45:54 asynchttp.go:288: Started job 69b4d15540c0951c129afef47d252af6 >[heketi] INFO 2018/08/19 13:45:54 Started async operation: Create Block Volume >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 200 OK in 149.52µs >[cmdexec] INFO 2018/08/19 13:45:54 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 43min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:45:54 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 49min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:45:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:45:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 200 OK in 152.268µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 40.633902ms >[asynchttp] INFO 2018/08/19 13:45:55 asynchttp.go:288: Started job fdb4b82100e9093d887c7a6143d57f1e >[heketi] INFO 2018/08/19 13:45:55 Started async operation: Create Block Volume >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 126.903µs >[cmdexec] INFO 2018/08/19 13:45:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 49min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:45:56 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 43min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:45:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 200 OK in 151.195µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 154.602µs >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 200 OK in 148.415µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 130.511µs >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 200 OK in 114.566µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 159.721µs >[kubeexec] DEBUG 2018/08/19 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_bea1593d93fb81194f81bca2d96d33ac ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:80bda231-8eb8-45ae-8be6-d2569b9ed07e", "USERNAME": "80bda231-8eb8-45ae-8be6-d2569b9ed07e", "PASSWORD": "dd7aeae5-890a-4de3-bb8a-6c0199c9ed77", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:45:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:45:59 asynchttp.go:292: Completed job 69b4d15540c0951c129afef47d252af6 in 4.436249584s >[kubeexec] DEBUG 2018/08/19 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:45:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:45:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 49min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/69b4d15540c0951c129afef47d252af6 >[negroni] Completed 303 See Other in 145.888µs >[negroni] Started GET /blockvolumes/bea1593d93fb81194f81bca2d96d33ac >[negroni] Completed 200 OK in 1.526121ms >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 119.654µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 116.341µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 129.016µs >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 200 OK in 131.415µs >[kubeexec] DEBUG 2018/08/19 13:46:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_acfec1da2a88b0ff8bdf848487ee527b ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:57c00225-48c3-42af-be7f-7a161c5a03c2", "USERNAME": "57c00225-48c3-42af-be7f-7a161c5a03c2", "PASSWORD": "7d82ea95-7a8a-482c-8495-0ed9a961b0db", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:46:03 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:46:03 asynchttp.go:292: Completed job fdb4b82100e9093d887c7a6143d57f1e in 7.62881254s >[negroni] Started GET /queue/fdb4b82100e9093d887c7a6143d57f1e >[negroni] Completed 303 See Other in 144.845µs >[negroni] Started GET /blockvolumes/acfec1da2a88b0ff8bdf848487ee527b >[negroni] Completed 200 OK in 2.57985ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 41.653267ms >[asynchttp] INFO 2018/08/19 13:46:25 asynchttp.go:288: Started job 1fd6424e5acd15d87e896111d61138b5 >[heketi] INFO 2018/08/19 13:46:25 Started async operation: Create Block Volume >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 200 OK in 131.635µs >[cmdexec] INFO 2018/08/19 13:46:25 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 43min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:46:25 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:46:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:46:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 200 OK in 114.82µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 33.929615ms >[asynchttp] INFO 2018/08/19 13:46:26 asynchttp.go:288: Started job 8f92a0b3c2c689f2a333686480a2b90d >[heketi] INFO 2018/08/19 13:46:26 Started async operation: Create Block Volume >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 120.506µs >[cmdexec] INFO 2018/08/19 13:46:26 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:46:26 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 43min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:46:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 200 OK in 130.963µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 143.117µs >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 200 OK in 143.864µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 115.425µs >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 200 OK in 238.714µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 172.225µs >[kubeexec] DEBUG 2018/08/19 13:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_db17218a10928b567a5497557a28fa07 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:216afa65-f603-463f-a502-47ac216f53c3", "USERNAME": "216afa65-f603-463f-a502-47ac216f53c3", "PASSWORD": "866878ce-07f0-4756-ba2d-a08fb1abbc83", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:46:29 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:46:29 asynchttp.go:292: Completed job 1fd6424e5acd15d87e896111d61138b5 in 4.449668037s >[kubeexec] DEBUG 2018/08/19 13:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:46:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1fd6424e5acd15d87e896111d61138b5 >[negroni] Completed 303 See Other in 159.709µs >[negroni] Started GET /blockvolumes/db17218a10928b567a5497557a28fa07 >[negroni] Completed 200 OK in 1.509871ms >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 124.572µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 184.417µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 158.965µs >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 200 OK in 137.355µs >[kubeexec] DEBUG 2018/08/19 13:46:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_ff21c104e0a7c0ac7635e6192c8e24a2 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bba07562-f149-4d3d-9851-d9b21941d0ac", "USERNAME": "bba07562-f149-4d3d-9851-d9b21941d0ac", "PASSWORD": "8b04e627-6780-4cdb-8fe1-d0a3e2d336a3", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:46:33 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:46:33 asynchttp.go:292: Completed job 8f92a0b3c2c689f2a333686480a2b90d in 7.405686532s >[negroni] Started GET /queue/8f92a0b3c2c689f2a333686480a2b90d >[negroni] Completed 303 See Other in 177.268µs >[negroni] Started GET /blockvolumes/ff21c104e0a7c0ac7635e6192c8e24a2 >[negroni] Completed 200 OK in 2.353209ms >[heketi] INFO 2018/08/19 13:46:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:46:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:46:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:46:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:46:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:46:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 44min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:46:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:46:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 38.259043ms >[asynchttp] INFO 2018/08/19 13:46:55 asynchttp.go:288: Started job 4061f21ef7967c9534d550ff8301c5c4 >[heketi] INFO 2018/08/19 13:46:55 Started async operation: Create Block Volume >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 200 OK in 113.513µs >[cmdexec] INFO 2018/08/19 13:46:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:46:55 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 44min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:46:55 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:46:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 200 OK in 187.989µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 37.066462ms >[asynchttp] INFO 2018/08/19 13:46:56 asynchttp.go:288: Started job f1be895a99bdf77cb1261a2b490c1c47 >[heketi] INFO 2018/08/19 13:46:56 Started async operation: Create Block Volume >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 152.191µs >[cmdexec] INFO 2018/08/19 13:46:56 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 44min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:46:56 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:46:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 50min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:46:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 200 OK in 187.881µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 143.838µs >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 200 OK in 180.05µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 124.582µs >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 200 OK in 186.709µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 124.41µs >[kubeexec] DEBUG 2018/08/19 13:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_4329277c5341fb577ed913cec4d8be0d ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e7efe99b-ad2b-4ad2-8aa7-af03f09394c4", "USERNAME": "e7efe99b-ad2b-4ad2-8aa7-af03f09394c4", "PASSWORD": "4fc91d5d-4fb7-4a58-b2e3-780cb12cafb5", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:47:00 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:47:00 asynchttp.go:292: Completed job 4061f21ef7967c9534d550ff8301c5c4 in 4.733086285s >[kubeexec] DEBUG 2018/08/19 13:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:47:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 50min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4061f21ef7967c9534d550ff8301c5c4 >[negroni] Completed 303 See Other in 140.829µs >[negroni] Started GET /blockvolumes/4329277c5341fb577ed913cec4d8be0d >[negroni] Completed 200 OK in 1.434727ms >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 170µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 188.834µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 196.85µs >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 200 OK in 135.978µs >[kubeexec] DEBUG 2018/08/19 13:47:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_b15b6706a9f68c4d207fc5756ab909d1 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fa0e8a18-4d4e-4063-ba54-11a9a6c3898c", "USERNAME": "fa0e8a18-4d4e-4063-ba54-11a9a6c3898c", "PASSWORD": "e579029f-b118-42b0-b82a-24496dd2c235", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:47:04 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:47:04 asynchttp.go:292: Completed job f1be895a99bdf77cb1261a2b490c1c47 in 7.856860356s >[negroni] Started GET /queue/f1be895a99bdf77cb1261a2b490c1c47 >[negroni] Completed 303 See Other in 122.921µs >[negroni] Started GET /blockvolumes/b15b6706a9f68c4d207fc5756ab909d1 >[negroni] Completed 200 OK in 2.836909ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 36.15899ms >[asynchttp] INFO 2018/08/19 13:47:25 asynchttp.go:288: Started job 6a3f0f64fa2e71b9eca67e222cff371e >[heketi] INFO 2018/08/19 13:47:25 Started async operation: Create Block Volume >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 200 OK in 93.707µs >[cmdexec] INFO 2018/08/19 13:47:25 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 44min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:47:25 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:47:26 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 51min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:47:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 200 OK in 138.858µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 36.474487ms >[asynchttp] INFO 2018/08/19 13:47:26 asynchttp.go:288: Started job 89d06b0b821858228f576f198bedd8fe >[heketi] INFO 2018/08/19 13:47:26 Started async operation: Create Block Volume >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 118.24µs >[cmdexec] INFO 2018/08/19 13:47:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 200 OK in 142.611µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 117.227µs >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 200 OK in 198.537µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 147.877µs >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 200 OK in 159.891µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 116.977µs >[kubeexec] DEBUG 2018/08/19 13:47:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_32e55d466a7751c61e5dd0ab2a6f9ac7 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bdb3135f-a11c-4d4d-a082-42533fef6c4a", "USERNAME": "bdb3135f-a11c-4d4d-a082-42533fef6c4a", "PASSWORD": "7d111fe6-bf3c-477f-bb43-3e3d9dc7f825", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:47:30 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:47:30 asynchttp.go:292: Completed job 6a3f0f64fa2e71b9eca67e222cff371e in 4.813787747s >[kubeexec] DEBUG 2018/08/19 13:47:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:47:30 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6a3f0f64fa2e71b9eca67e222cff371e >[negroni] Completed 303 See Other in 114.253µs >[negroni] Started GET /blockvolumes/32e55d466a7751c61e5dd0ab2a6f9ac7 >[negroni] Completed 200 OK in 2.019772ms >[kubeexec] DEBUG 2018/08/19 13:47:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 44min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:47:30 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 138.21µs >[kubeexec] DEBUG 2018/08/19 13:47:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 51min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:47:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 191.733µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 140.647µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 168.474µs >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 200 OK in 133.605µs >[kubeexec] DEBUG 2018/08/19 13:47:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_09974f2668bda0bc3c6bd81179ee5115 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d99107e5-3896-4c46-8e73-56b3a1b7acbd", "USERNAME": "d99107e5-3896-4c46-8e73-56b3a1b7acbd", "PASSWORD": "afaa1d41-93fc-49c9-ace7-a4bae202a91c", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:47:35 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:47:35 asynchttp.go:292: Completed job 89d06b0b821858228f576f198bedd8fe in 8.369004369s >[negroni] Started GET /queue/89d06b0b821858228f576f198bedd8fe >[negroni] Completed 303 See Other in 131.576µs >[negroni] Started GET /blockvolumes/09974f2668bda0bc3c6bd81179ee5115 >[negroni] Completed 200 OK in 2.372922ms >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 43.440626ms >[asynchttp] INFO 2018/08/19 13:47:56 asynchttp.go:288: Started job 66a4fbe718b4224a30a98589b384c9b9 >[heketi] INFO 2018/08/19 13:47:56 Started async operation: Create Block Volume >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 200 OK in 113.129µs >[cmdexec] INFO 2018/08/19 13:47:56 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 51min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:47:56 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 45min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:47:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:47:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 200 OK in 199.167µs >[negroni] Started POST /blockvolumes >[negroni] Completed 202 Accepted in 40.637904ms >[asynchttp] INFO 2018/08/19 13:47:57 asynchttp.go:288: Started job 886ae406b50614e271364dffd16f2fb8 >[heketi] INFO 2018/08/19 13:47:57 Started async operation: Create Block Volume >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 80.309µs >[cmdexec] INFO 2018/08/19 13:47:57 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 45min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:47:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:47:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 51min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:47:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 200 OK in 117.825µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 154.285µs >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 200 OK in 124.526µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 124.373µs >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 200 OK in 116.863µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 228.234µs >[kubeexec] DEBUG 2018/08/19 13:48:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create blkhosting/blockvol_fb6a139c54dac70f1e392e3ebee5112f ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b9f57e8f-dc26-406f-bd60-0e55b8168029", "USERNAME": "b9f57e8f-dc26-406f-bd60-0e55b8168029", "PASSWORD": "273dc69f-527c-488c-94ae-5f876e0947ac", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:48:01 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:48:01 asynchttp.go:292: Completed job 66a4fbe718b4224a30a98589b384c9b9 in 4.884599359s >[negroni] Started GET /queue/66a4fbe718b4224a30a98589b384c9b9 >[negroni] Completed 303 See Other in 152.735µs >[negroni] Started GET /blockvolumes/fb6a139c54dac70f1e392e3ebee5112f >[negroni] Completed 200 OK in 2.736633ms >[kubeexec] DEBUG 2018/08/19 13:48:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:48:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 130.939µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 198.854µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 136.815µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 132.074µs >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 200 OK in 134.493µs >[kubeexec] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create blkhosting/blockvol_cf53b5b9bea386daa599adad0df58a3d ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "IQN": "iqn.2016-12.org.gluster-block:4be4e135-1e52-4ba5-8e2b-07cc93158a0c", "USERNAME": "4be4e135-1e52-4ba5-8e2b-07cc93158a0c", "PASSWORD": "a4bb31a0-6fa4-41b5-9b7e-9f8c8f897a7d", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "ROLLBACK FAILED ON": [ "10.70.46.152", "10.70.47.54", "10.70.47.183" ], "RESULT": "FAIL" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete blkhosting/blockvol_cf53b5b9bea386daa599adad0df58a3d --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "FAILED ON": [ "10.70.46.152" ], "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_cf53b5b9bea386daa599adad0df58a3d: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 13:48:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 51min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete blkhosting/blockvol_cf53b5b9bea386daa599adad0df58a3d --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "FAILED ON": [ "10.70.46.152" ], "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_cf53b5b9bea386daa599adad0df58a3d: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 13:48:05 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 13:48:05 asynchttp.go:292: Completed job 886ae406b50614e271364dffd16f2fb8 in 8.602976391s >[negroni] Started GET /queue/886ae406b50614e271364dffd16f2fb8 >[negroni] Completed 500 Internal Server Error in 158.379µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:48:26 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 19.808639ms >[asynchttp] INFO 2018/08/19 13:48:26 asynchttp.go:288: Started job ae03eb1966522fb02e2a3045f395748e >[heketi] INFO 2018/08/19 13:48:26 Started async operation: Create Block Volume >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 92.878µs >[cmdexec] INFO 2018/08/19 13:48:26 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:48:26 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 45min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:48:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:48:26 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 131.926µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:48:27 Free size 0 is less than the requested block volume size 1 >[asynchttp] INFO 2018/08/19 13:48:27 asynchttp.go:288: Started job c227c3bd8b53c7e3b9ae2342daf8a9e3 >[heketi] INFO 2018/08/19 13:48:27 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 35.690154ms >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 112.391µs >[cmdexec] INFO 2018/08/19 13:48:27 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 45min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:48:27 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:48:27 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 190.406µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 160.697µs >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 134.147µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 120.673µs >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 174.346µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 179.876µs >[kubeexec] DEBUG 2018/08/19 13:48:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_56e188b9b483292e6b85d712e68fdbb4 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6ce4268a-5766-4759-85fe-ed4f5ab9b897", "USERNAME": "6ce4268a-5766-4759-85fe-ed4f5ab9b897", "PASSWORD": "d79ccec9-5679-408b-b2db-0059cb6455fc", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 200 OK in 151.175µs >[heketi] INFO 2018/08/19 13:48:31 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:48:31 asynchttp.go:292: Completed job ae03eb1966522fb02e2a3045f395748e in 5.036903609s >[kubeexec] DEBUG 2018/08/19 13:48:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:48:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 128.439µs >[kubeexec] DEBUG 2018/08/19 13:48:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ae03eb1966522fb02e2a3045f395748e >[negroni] Completed 303 See Other in 196.064µs >[negroni] Started GET /blockvolumes/56e188b9b483292e6b85d712e68fdbb4 >[negroni] Completed 200 OK in 1.89175ms >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 186.56µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 129.871µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 205.499µs >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 200 OK in 126.546µs >[kubeexec] DEBUG 2018/08/19 13:48:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c6f32c4e3dc6d93412c3c799c0545265 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6930b723-4f52-4951-8e53-a238910784c0", "USERNAME": "6930b723-4f52-4951-8e53-a238910784c0", "PASSWORD": "c67d1d26-e64e-4118-9270-65a99e470e8c", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:48:35 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:48:35 asynchttp.go:292: Completed job c227c3bd8b53c7e3b9ae2342daf8a9e3 in 8.279806811s >[negroni] Started GET /queue/c227c3bd8b53c7e3b9ae2342daf8a9e3 >[negroni] Completed 303 See Other in 142.084µs >[negroni] Started GET /blockvolumes/c6f32c4e3dc6d93412c3c799c0545265 >[negroni] Completed 200 OK in 2.64616ms >[heketi] INFO 2018/08/19 13:48:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:48:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:48:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:48:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:48:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:48:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 46min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:48:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:48:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:48:56 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 16.652197ms >[asynchttp] INFO 2018/08/19 13:48:56 asynchttp.go:288: Started job bb6bd5dc80003e0a3ccc78a18a0b44f5 >[heketi] INFO 2018/08/19 13:48:56 Started async operation: Create Block Volume >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 200 OK in 144.62µs >[cmdexec] INFO 2018/08/19 13:48:56 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 46min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:48:56 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:48:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:48:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:48:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 200 OK in 199.227µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:48:58 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 20.10407ms >[asynchttp] INFO 2018/08/19 13:48:58 asynchttp.go:288: Started job 83af072fcb7ad6b3d54aaa91b565660f >[heketi] INFO 2018/08/19 13:48:58 Started async operation: Create Block Volume >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 137.917µs >[cmdexec] INFO 2018/08/19 13:48:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 200 OK in 156.244µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 133.977µs >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 200 OK in 179.996µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 147.563µs >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 200 OK in 123.45µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 136.066µs >[kubeexec] DEBUG 2018/08/19 13:49:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_88f00938624c62842fd4f86908fdf3e7 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:68f0fee7-56d3-46ae-965a-f9163cb1a29c", "USERNAME": "68f0fee7-56d3-46ae-965a-f9163cb1a29c", "PASSWORD": "200ce46a-cba9-4920-9717-3732369aec42", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:49:01 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:49:01 asynchttp.go:292: Completed job bb6bd5dc80003e0a3ccc78a18a0b44f5 in 4.991053033s >[negroni] Started GET /queue/bb6bd5dc80003e0a3ccc78a18a0b44f5 >[negroni] Completed 303 See Other in 111.718µs >[negroni] Started GET /blockvolumes/88f00938624c62842fd4f86908fdf3e7 >[negroni] Completed 200 OK in 2.632539ms >[kubeexec] DEBUG 2018/08/19 13:49:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:49:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 130.007µs >[kubeexec] DEBUG 2018/08/19 13:49:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 52min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:49:02 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 46min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:49:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 52min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 210.159µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 139.966µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 122.957µs >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 200 OK in 187.486µs >[kubeexec] DEBUG 2018/08/19 13:49:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5a06181c47393c1640e156652e19542b ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f5668aec-eb94-4fa5-805b-94759b56a0cb", "USERNAME": "f5668aec-eb94-4fa5-805b-94759b56a0cb", "PASSWORD": "b5cf6463-9244-456c-bbeb-c8747d9495b7", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:49:06 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:49:06 asynchttp.go:292: Completed job 83af072fcb7ad6b3d54aaa91b565660f in 8.676708669s >[negroni] Started GET /queue/83af072fcb7ad6b3d54aaa91b565660f >[negroni] Completed 303 See Other in 131.334µs >[negroni] Started GET /blockvolumes/5a06181c47393c1640e156652e19542b >[negroni] Completed 200 OK in 2.32578ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:49:27 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 19.979586ms >[asynchttp] INFO 2018/08/19 13:49:27 asynchttp.go:288: Started job faf2de38a297cd4cd3022c902f063021 >[heketi] INFO 2018/08/19 13:49:27 Started async operation: Create Block Volume >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 200 OK in 116.508µs >[cmdexec] INFO 2018/08/19 13:49:27 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 46min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:49:27 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 53min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:49:27 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:49:27 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 200 OK in 152.723µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:49:28 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 16.178458ms >[asynchttp] INFO 2018/08/19 13:49:28 asynchttp.go:288: Started job cab9231b85cc1f3932c7b463bb9b8cd6 >[heketi] INFO 2018/08/19 13:49:28 Started async operation: Create Block Volume >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 130.497µs >[cmdexec] INFO 2018/08/19 13:49:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 46min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:49:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 200 OK in 129.684µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 145.504µs >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 200 OK in 129.866µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 149.229µs >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 200 OK in 214.972µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 170.121µs >[kubeexec] DEBUG 2018/08/19 13:49:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_967e3e6f6fe75171b60ac8622787c214 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4b1d943f-8770-440f-9725-5c74fa4ddc7d", "USERNAME": "4b1d943f-8770-440f-9725-5c74fa4ddc7d", "PASSWORD": "fb434216-2c6b-4237-a386-a1f350b21267", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:49:31 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:49:31 asynchttp.go:292: Completed job faf2de38a297cd4cd3022c902f063021 in 4.818733686s >[kubeexec] DEBUG 2018/08/19 13:49:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:49:32 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/faf2de38a297cd4cd3022c902f063021 >[negroni] Completed 303 See Other in 147.225µs >[negroni] Started GET /blockvolumes/967e3e6f6fe75171b60ac8622787c214 >[negroni] Completed 200 OK in 2.088883ms >[kubeexec] DEBUG 2018/08/19 13:49:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 53min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:49:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 168.25µs >[kubeexec] DEBUG 2018/08/19 13:49:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 136.397µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 124.695µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 200.147µs >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 200 OK in 167.996µs >[kubeexec] DEBUG 2018/08/19 13:49:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c1154e09ac7789a70e75ac652c2049aa ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:9987a985-8f33-4fe7-909e-474b66fcd949", "USERNAME": "9987a985-8f33-4fe7-909e-474b66fcd949", "PASSWORD": "0850636a-622f-432a-ad57-6d32f2704d99", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:49:36 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:49:36 asynchttp.go:292: Completed job cab9231b85cc1f3932c7b463bb9b8cd6 in 8.410990165s >[negroni] Started GET /queue/cab9231b85cc1f3932c7b463bb9b8cd6 >[negroni] Completed 303 See Other in 151.449µs >[negroni] Started GET /blockvolumes/c1154e09ac7789a70e75ac652c2049aa >[negroni] Completed 200 OK in 2.212037ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:49:57 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 20.316249ms >[asynchttp] INFO 2018/08/19 13:49:57 asynchttp.go:288: Started job 1d1407f5303b39d8bde4996948a15f32 >[heketi] INFO 2018/08/19 13:49:57 Started async operation: Create Block Volume >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 200 OK in 124.885µs >[cmdexec] INFO 2018/08/19 13:49:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:49:57 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 53min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:49:57 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 47min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:49:57 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 200 OK in 167.473µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:49:58 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 17.169536ms >[asynchttp] INFO 2018/08/19 13:49:58 asynchttp.go:288: Started job 71d1b60d61e30083ed034897617e3915 >[heketi] INFO 2018/08/19 13:49:58 Started async operation: Create Block Volume >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 89.099µs >[cmdexec] INFO 2018/08/19 13:49:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 47min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:49:58 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:49:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 53min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:49:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 200 OK in 199.341µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 250.993µs >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 200 OK in 124.268µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 133.5µs >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 200 OK in 135.373µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 154.567µs >[kubeexec] DEBUG 2018/08/19 13:50:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_a48b38a942e50c983c112507a827a10b ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:48105f7d-914f-4c1c-a936-7389eeb4f53b", "USERNAME": "48105f7d-914f-4c1c-a936-7389eeb4f53b", "PASSWORD": "438e65e3-9a3a-4619-bc26-eb225c3a0656", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:50:02 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:50:02 asynchttp.go:292: Completed job 1d1407f5303b39d8bde4996948a15f32 in 4.87240476s >[kubeexec] DEBUG 2018/08/19 13:50:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:50:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1d1407f5303b39d8bde4996948a15f32 >[negroni] Completed 303 See Other in 139.78µs >[negroni] Started GET /blockvolumes/a48b38a942e50c983c112507a827a10b >[negroni] Completed 200 OK in 1.382102ms >[kubeexec] DEBUG 2018/08/19 13:50:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 53min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 111.514µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 175.13µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 153.674µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 126.153µs >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 200 OK in 155.877µs >[kubeexec] DEBUG 2018/08/19 13:50:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_65c2342d50cffe6d60f8554b48f263bb ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:9d29c77f-6eb4-47e8-95b9-e7553c3a4e34", "USERNAME": "9d29c77f-6eb4-47e8-95b9-e7553c3a4e34", "PASSWORD": "00562b42-bbb7-4feb-a2cc-62f130952a7d", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:50:07 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:50:07 asynchttp.go:292: Completed job 71d1b60d61e30083ed034897617e3915 in 8.707482349s >[negroni] Started GET /queue/71d1b60d61e30083ed034897617e3915 >[negroni] Completed 303 See Other in 138.956µs >[negroni] Started GET /blockvolumes/65c2342d50cffe6d60f8554b48f263bb >[negroni] Completed 200 OK in 2.86755ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:50:27 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.650872ms >[asynchttp] INFO 2018/08/19 13:50:27 asynchttp.go:288: Started job 30de584e774ecb350c314291c0606b4f >[heketi] INFO 2018/08/19 13:50:27 Started async operation: Create Block Volume >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 200 OK in 110.464µs >[cmdexec] INFO 2018/08/19 13:50:27 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:50:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:50:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 47min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:50:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 200 OK in 140.734µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:50:29 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 21.529829ms >[asynchttp] INFO 2018/08/19 13:50:29 asynchttp.go:288: Started job ac01d26b044643e26e7890f0227d38e4 >[heketi] INFO 2018/08/19 13:50:29 Started async operation: Create Block Volume >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 183.152µs >[cmdexec] INFO 2018/08/19 13:50:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 200 OK in 190.422µs >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 174.925µs >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 200 OK in 209.273µs >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 135.633µs >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 200 OK in 152.88µs >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 243.287µs >[kubeexec] DEBUG 2018/08/19 13:50:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ff151940825b87dddea56e4d27dbcef0 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a69a029f-93e9-40be-b134-cc9542928652", "USERNAME": "a69a029f-93e9-40be-b134-cc9542928652", "PASSWORD": "0409eb33-8084-439f-8012-e576c4429545", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:50:32 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:50:32 asynchttp.go:292: Completed job 30de584e774ecb350c314291c0606b4f in 4.988604485s >[negroni] Started GET /queue/30de584e774ecb350c314291c0606b4f >[negroni] Completed 303 See Other in 156.464µs >[negroni] Started GET /blockvolumes/ff151940825b87dddea56e4d27dbcef0 >[negroni] Completed 200 OK in 3.177484ms >[kubeexec] DEBUG 2018/08/19 13:50:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:50:33 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 119.567µs >[kubeexec] DEBUG 2018/08/19 13:50:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 47min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:50:33 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:50:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 129.428µs >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 188.041µs >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 122.485µs >[heketi] INFO 2018/08/19 13:50:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:50:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 200 OK in 122.35µs >[kubeexec] DEBUG 2018/08/19 13:50:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ca1cddbfd0b6adeeb47798d2f057b758 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4b26dc84-3f2e-42e3-b0ae-2db3277aa56c", "USERNAME": "4b26dc84-3f2e-42e3-b0ae-2db3277aa56c", "PASSWORD": "163485a1-bf35-4735-91fa-fe3327745938", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:50:37 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:50:37 asynchttp.go:292: Completed job ac01d26b044643e26e7890f0227d38e4 in 8.834830787s >[kubeexec] DEBUG 2018/08/19 13:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:50:38 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:50:38 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ac01d26b044643e26e7890f0227d38e4 >[negroni] Completed 303 See Other in 143.362µs >[negroni] Started GET /blockvolumes/ca1cddbfd0b6adeeb47798d2f057b758 >[negroni] Completed 200 OK in 2.759754ms >[kubeexec] DEBUG 2018/08/19 13:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:50:38 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:50:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 48min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:50:38 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:50:38 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:50:58 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 20.679077ms >[asynchttp] INFO 2018/08/19 13:50:58 asynchttp.go:288: Started job 2872b02a4739b04f9548e2120b9ecf2d >[heketi] INFO 2018/08/19 13:50:58 Started async operation: Create Block Volume >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 119.605µs >[cmdexec] INFO 2018/08/19 13:50:58 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 48min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:50:58 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:50:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:50:58 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:58 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 121.443µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:50:59 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.298817ms >[asynchttp] INFO 2018/08/19 13:50:59 asynchttp.go:288: Started job bd8d05e9bb9ae566ed447fe759d550b2 >[heketi] INFO 2018/08/19 13:50:59 Started async operation: Create Block Volume >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 101.547µs >[cmdexec] INFO 2018/08/19 13:50:59 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 48min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:50:59 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:50:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 54min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:50:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 180.552µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 129.947µs >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 214.667µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 187.847µs >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 158.297µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 187.511µs >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 200 OK in 197.091µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 198.549µs >[kubeexec] DEBUG 2018/08/19 13:51:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_3acffaf6f8d72c76930cadab288565a9 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c9053c07-7a8f-47c4-b384-1a12236f9292", "USERNAME": "c9053c07-7a8f-47c4-b384-1a12236f9292", "PASSWORD": "a9c62121-08e1-4547-9379-ef30aae1814e", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:51:03 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:51:03 asynchttp.go:292: Completed job 2872b02a4739b04f9548e2120b9ecf2d in 5.38802705s >[kubeexec] DEBUG 2018/08/19 13:51:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:51:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 54min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2872b02a4739b04f9548e2120b9ecf2d >[negroni] Completed 303 See Other in 139.064µs >[negroni] Started GET /blockvolumes/3acffaf6f8d72c76930cadab288565a9 >[negroni] Completed 200 OK in 1.862338ms >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 128.65µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 178.234µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 137.895µs >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 226.973µs >[kubeexec] DEBUG 2018/08/19 13:51:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_1870e3ac7cfe3e7239a426d36b8df6fb ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4dbb160d-3731-414c-817e-832f6ed9c1e8", "USERNAME": "4dbb160d-3731-414c-817e-832f6ed9c1e8", "PASSWORD": "04734fed-aec0-4b0c-9176-5f511105aad3", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 200 OK in 130.318µs >[heketi] INFO 2018/08/19 13:51:08 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:51:08 asynchttp.go:292: Completed job bd8d05e9bb9ae566ed447fe759d550b2 in 9.062611857s >[negroni] Started GET /queue/bd8d05e9bb9ae566ed447fe759d550b2 >[negroni] Completed 303 See Other in 172.36µs >[negroni] Started GET /blockvolumes/1870e3ac7cfe3e7239a426d36b8df6fb >[negroni] Completed 200 OK in 2.385594ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:51:28 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 22.615742ms >[asynchttp] INFO 2018/08/19 13:51:28 asynchttp.go:288: Started job 9c772a3a440ff36edac6d02aa0180e01 >[heketi] INFO 2018/08/19 13:51:28 Started async operation: Create Block Volume >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 94.169µs >[cmdexec] INFO 2018/08/19 13:51:28 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 48min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:51:28 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:51:28 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 55min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:51:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 138.28µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:51:29 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 20.280561ms >[asynchttp] INFO 2018/08/19 13:51:29 asynchttp.go:288: Started job 89bf82f78fb80b9cc764d3356e7d9576 >[heketi] INFO 2018/08/19 13:51:29 Started async operation: Create Block Volume >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 142.53µs >[cmdexec] INFO 2018/08/19 13:51:29 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 48min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:51:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 145.177µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 129.39µs >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 137.576µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 130.427µs >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 145.337µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 122.693µs >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 200 OK in 123.171µs >[kubeexec] DEBUG 2018/08/19 13:51:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_a3f5fdec254f4ec58d73f612c01a096f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4430fdbd-e1b4-49d6-84e3-283f3ac83fc1", "USERNAME": "4430fdbd-e1b4-49d6-84e3-283f3ac83fc1", "PASSWORD": "eb76cfcb-585f-47a5-a38f-1a0e8106cdad", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:51:33 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:51:33 asynchttp.go:292: Completed job 9c772a3a440ff36edac6d02aa0180e01 in 5.07808506s >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 124.047µs >[kubeexec] DEBUG 2018/08/19 13:51:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:51:33 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 55min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:51:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9c772a3a440ff36edac6d02aa0180e01 >[negroni] Completed 303 See Other in 147.03µs >[negroni] Started GET /blockvolumes/a3f5fdec254f4ec58d73f612c01a096f >[negroni] Completed 200 OK in 1.31095ms >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 143.876µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 120.588µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 116.416µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 140.909µs >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 200 OK in 159.548µs >[kubeexec] DEBUG 2018/08/19 13:51:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_dd7247cd73f1daa3f3f34e40a1f9b68a ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:adf3e145-f018-4fb5-8c2d-bb8229269ea3", "USERNAME": "adf3e145-f018-4fb5-8c2d-bb8229269ea3", "PASSWORD": "cbd87cc0-0819-4fdf-8f90-43df12b99bfc", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:51:39 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:51:39 asynchttp.go:292: Completed job 89bf82f78fb80b9cc764d3356e7d9576 in 9.392729801s >[negroni] Started GET /queue/89bf82f78fb80b9cc764d3356e7d9576 >[negroni] Completed 303 See Other in 181.827µs >[negroni] Started GET /blockvolumes/dd7247cd73f1daa3f3f34e40a1f9b68a >[negroni] Completed 200 OK in 2.535279ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:51:59 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.851429ms >[asynchttp] INFO 2018/08/19 13:51:59 asynchttp.go:288: Started job ab0613a95f0d8638f9c8aa98f00bdcf7 >[heketi] INFO 2018/08/19 13:51:59 Started async operation: Create Block Volume >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 117.127µs >[cmdexec] INFO 2018/08/19 13:51:59 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 49min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:51:59 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 55min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:51:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:51:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:51:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 136.544µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:52:00 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.09991ms >[asynchttp] INFO 2018/08/19 13:52:00 asynchttp.go:288: Started job dcca85097770ba80df60ba9ba146a71e >[heketi] INFO 2018/08/19 13:52:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 114.399µs >[cmdexec] INFO 2018/08/19 13:52:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 129.526µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 129.168µs >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 173.712µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 123.882µs >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 138.681µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 125.256µs >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 200 OK in 131.578µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 124.25µs >[kubeexec] DEBUG 2018/08/19 13:52:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f1fe875d53d1facb16d92101d8bf7268 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:17f4cef4-2353-4b79-b4c1-edb0e2f53ecc", "USERNAME": "17f4cef4-2353-4b79-b4c1-edb0e2f53ecc", "PASSWORD": "e7c643c4-f803-490a-8d33-9bc7be2c43f9", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:52:04 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:52:04 asynchttp.go:292: Completed job ab0613a95f0d8638f9c8aa98f00bdcf7 in 5.28662916s >[kubeexec] DEBUG 2018/08/19 13:52:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:52:04 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 55min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:52:04 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 49min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:52:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 55min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ab0613a95f0d8638f9c8aa98f00bdcf7 >[negroni] Completed 303 See Other in 129.988µs >[negroni] Started GET /blockvolumes/f1fe875d53d1facb16d92101d8bf7268 >[negroni] Completed 200 OK in 1.430197ms >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 182.017µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 164.145µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 119.205µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 153.103µs >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 200 OK in 191.609µs >[kubeexec] DEBUG 2018/08/19 13:52:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_dff48a87c5c1b598a8f2b977eaaa2253 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c3f22d46-9e05-40fd-8c81-41bfc294fc3d", "USERNAME": "c3f22d46-9e05-40fd-8c81-41bfc294fc3d", "PASSWORD": "7276eb14-1c1b-4aad-91da-860ba3b65718", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:52:09 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:52:09 asynchttp.go:292: Completed job dcca85097770ba80df60ba9ba146a71e in 9.354308649s >[negroni] Started GET /queue/dcca85097770ba80df60ba9ba146a71e >[negroni] Completed 303 See Other in 160.895µs >[negroni] Started GET /blockvolumes/dff48a87c5c1b598a8f2b977eaaa2253 >[negroni] Completed 200 OK in 2.224908ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:52:29 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.074694ms >[asynchttp] INFO 2018/08/19 13:52:29 asynchttp.go:288: Started job 9a70fb8e80802cfe01d8e50afbcae13d >[heketi] INFO 2018/08/19 13:52:29 Started async operation: Create Block Volume >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 123.67µs >[cmdexec] INFO 2018/08/19 13:52:29 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:52:29 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 49min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:52:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:52:29 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 149.287µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:52:30 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 20.724872ms >[asynchttp] INFO 2018/08/19 13:52:30 asynchttp.go:288: Started job 78ed76afd0f08c71c73bf50546bcfc07 >[heketi] INFO 2018/08/19 13:52:30 Started async operation: Create Block Volume >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 124.357µs >[cmdexec] INFO 2018/08/19 13:52:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 128.006µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 133.848µs >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 125.697µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 178.894µs >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 147.123µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 176.16µs >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 200 OK in 147.58µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 151.319µs >[kubeexec] DEBUG 2018/08/19 13:52:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_b1de52b410d88d506db119f1b987324b ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7655b687-f3f0-4d25-b6aa-52355641fc29", "USERNAME": "7655b687-f3f0-4d25-b6aa-52355641fc29", "PASSWORD": "33a0be0f-4582-4afa-a889-f88033adb325", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:52:34 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:52:34 asynchttp.go:292: Completed job 9a70fb8e80802cfe01d8e50afbcae13d in 5.360185919s >[kubeexec] DEBUG 2018/08/19 13:52:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:52:34 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:52:35 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 50min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:52:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9a70fb8e80802cfe01d8e50afbcae13d >[negroni] Completed 303 See Other in 152.091µs >[kubeexec] DEBUG 2018/08/19 13:52:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /blockvolumes/b1de52b410d88d506db119f1b987324b >[negroni] Completed 200 OK in 1.627398ms >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 162.214µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 203.606µs >[heketi] INFO 2018/08/19 13:52:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:52:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 139.174µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 135.652µs >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 200 OK in 153.105µs >[kubeexec] DEBUG 2018/08/19 13:52:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_212db7746a0cbf9b5416a6bb180fe735 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7394b53c-0fac-4908-8436-f98a7a7fa086", "USERNAME": "7394b53c-0fac-4908-8436-f98a7a7fa086", "PASSWORD": "5ef90690-6042-4891-963e-6862446a1d02", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:52:40 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:52:40 asynchttp.go:292: Completed job 78ed76afd0f08c71c73bf50546bcfc07 in 9.700535548s >[kubeexec] DEBUG 2018/08/19 13:52:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:52:40 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:52:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:52:40 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:52:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/78ed76afd0f08c71c73bf50546bcfc07 >[negroni] Completed 303 See Other in 152.935µs >[negroni] Started GET /blockvolumes/212db7746a0cbf9b5416a6bb180fe735 >[negroni] Completed 200 OK in 2.756605ms >[kubeexec] DEBUG 2018/08/19 13:52:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 50min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:52:40 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:52:40 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:52:59 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 32.287846ms >[asynchttp] INFO 2018/08/19 13:52:59 asynchttp.go:288: Started job c0e3f2953eabd04522df8c1ba659a264 >[heketi] INFO 2018/08/19 13:52:59 Started async operation: Create Block Volume >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 123.872µs >[cmdexec] INFO 2018/08/19 13:52:59 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:52:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 50min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:52:59 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:53:00 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:53:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 156.35µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:53:00 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 21.803007ms >[asynchttp] INFO 2018/08/19 13:53:00 asynchttp.go:288: Started job b2fe9ead58d45cfb16609332d0e3169f >[heketi] INFO 2018/08/19 13:53:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 109.346µs >[cmdexec] INFO 2018/08/19 13:53:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 146.982µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 147.776µs >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 140.275µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 138.326µs >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 122.369µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 147.12µs >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 200 OK in 145.104µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 151.307µs >[kubeexec] DEBUG 2018/08/19 13:53:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_93243bb345079d97821445cedb4e37cc ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d16cae97-791e-402e-8a9b-9e84691c7c8b", "USERNAME": "d16cae97-791e-402e-8a9b-9e84691c7c8b", "PASSWORD": "3f863853-e20d-4e3d-9a2c-aa3467b8a2ca", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:53:05 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:53:05 asynchttp.go:292: Completed job c0e3f2953eabd04522df8c1ba659a264 in 5.355246874s >[kubeexec] DEBUG 2018/08/19 13:53:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:53:05 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 56min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:53:05 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 50min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:53:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 56min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c0e3f2953eabd04522df8c1ba659a264 >[negroni] Completed 303 See Other in 160.78µs >[negroni] Started GET /blockvolumes/93243bb345079d97821445cedb4e37cc >[negroni] Completed 200 OK in 1.556632ms >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 169.974µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 194.619µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 175.05µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 113.337µs >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 200 OK in 126.526µs >[kubeexec] DEBUG 2018/08/19 13:53:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_b3ef7efe8e339b961055cdb3f85812ca ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3befd286-0677-42bc-a7cc-d55bdbab234f", "USERNAME": "3befd286-0677-42bc-a7cc-d55bdbab234f", "PASSWORD": "fe9d392f-2a5e-4609-b52c-f1ca08fd2ef7", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:53:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:53:10 asynchttp.go:292: Completed job b2fe9ead58d45cfb16609332d0e3169f in 9.71250055s >[negroni] Started GET /queue/b2fe9ead58d45cfb16609332d0e3169f >[negroni] Completed 303 See Other in 181.221µs >[negroni] Started GET /blockvolumes/b3ef7efe8e339b961055cdb3f85812ca >[negroni] Completed 200 OK in 2.734792ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:53:30 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 29.586946ms >[asynchttp] INFO 2018/08/19 13:53:30 asynchttp.go:288: Started job 063440ad543ced0dff5a921bceceb115 >[heketi] INFO 2018/08/19 13:53:30 Started async operation: Create Block Volume >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 120.043µs >[cmdexec] INFO 2018/08/19 13:53:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:53:30 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 57min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:53:30 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 50min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:53:30 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:30 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 174.566µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:53:31 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 30.263589ms >[asynchttp] INFO 2018/08/19 13:53:31 asynchttp.go:288: Started job 8235052ac6eca33ec11b5706b4f45bd5 >[heketi] INFO 2018/08/19 13:53:31 Started async operation: Create Block Volume >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 153.617µs >[cmdexec] INFO 2018/08/19 13:53:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 136.902µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 119.018µs >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 150.257µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 165.155µs >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 128.092µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 152.617µs >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 200 OK in 147.23µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 129.868µs >[kubeexec] DEBUG 2018/08/19 13:53:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e6a5aef3113e227061537dd558125930 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:4f655ff5-e6bc-4d21-bce3-9eb414f2d41f", "USERNAME": "4f655ff5-e6bc-4d21-bce3-9eb414f2d41f", "PASSWORD": "a51f577b-d2ef-4d73-9409-2f0826eab51a", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:53:35 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:53:35 asynchttp.go:292: Completed job 063440ad543ced0dff5a921bceceb115 in 5.383390083s >[kubeexec] DEBUG 2018/08/19 13:53:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:53:35 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 57min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:53:35 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 51min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:53:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:53:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/063440ad543ced0dff5a921bceceb115 >[negroni] Completed 303 See Other in 152.859µs >[negroni] Started GET /blockvolumes/e6a5aef3113e227061537dd558125930 >[negroni] Completed 200 OK in 2.400706ms >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 124.285µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 134.328µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 164.091µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 165.329µs >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 200 OK in 122.214µs >[kubeexec] DEBUG 2018/08/19 13:53:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5469601edf305a8bf2094b0e9062d04b ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e08b1f57-104b-4eba-b324-c3c6c628fc7b", "USERNAME": "e08b1f57-104b-4eba-b324-c3c6c628fc7b", "PASSWORD": "6fe3974b-05d8-46d2-b4a9-0c1d9a3adf7f", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:53:40 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:53:40 asynchttp.go:292: Completed job 8235052ac6eca33ec11b5706b4f45bd5 in 9.677423112s >[negroni] Started GET /queue/8235052ac6eca33ec11b5706b4f45bd5 >[negroni] Completed 303 See Other in 171.962µs >[negroni] Started GET /blockvolumes/5469601edf305a8bf2094b0e9062d04b >[negroni] Completed 200 OK in 2.68215ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:54:00 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.724582ms >[asynchttp] INFO 2018/08/19 13:54:00 asynchttp.go:288: Started job 43520a87c869b67f6090892dda80303c >[heketi] INFO 2018/08/19 13:54:00 Started async operation: Create Block Volume >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 125.948µs >[cmdexec] INFO 2018/08/19 13:54:00 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 57min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:54:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:54:00 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 51min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:54:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 189.992µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:54:01 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.596925ms >[asynchttp] INFO 2018/08/19 13:54:01 asynchttp.go:288: Started job 16158705cbb9d71775d248f7f9c8508b >[heketi] INFO 2018/08/19 13:54:01 Started async operation: Create Block Volume >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 147.243µs >[cmdexec] INFO 2018/08/19 13:54:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 163.742µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 199.892µs >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 142.769µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 151.193µs >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 118.006µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 157.925µs >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 200 OK in 136.261µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 124.273µs >[kubeexec] DEBUG 2018/08/19 13:54:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fe0299166eb06f4b44ffe352557cb192 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:026c5ce7-fa66-4480-a668-e6a6246f2027", "USERNAME": "026c5ce7-fa66-4480-a668-e6a6246f2027", "PASSWORD": "e25a1867-2576-4086-b39d-2b905c47a1a4", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:54:06 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:54:06 asynchttp.go:292: Completed job 43520a87c869b67f6090892dda80303c in 5.792137374s >[kubeexec] DEBUG 2018/08/19 13:54:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:54:06 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 57min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:54:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/43520a87c869b67f6090892dda80303c >[negroni] Completed 303 See Other in 161.243µs >[negroni] Started GET /blockvolumes/fe0299166eb06f4b44ffe352557cb192 >[negroni] Completed 200 OK in 2.787552ms >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 124.181µs >[kubeexec] DEBUG 2018/08/19 13:54:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 51min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:54:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 57min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 128.014µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 171.196µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 114.762µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 176.715µs >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 200 OK in 425.797µs >[kubeexec] DEBUG 2018/08/19 13:54:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_192df25125cd0f660bd360b8c8b20d14 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:bf89a03f-7428-468d-9113-a1988c6bf6c0", "USERNAME": "bf89a03f-7428-468d-9113-a1988c6bf6c0", "PASSWORD": "fb80ed95-9e4e-4878-9890-a52016286a80", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:54:12 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:54:12 asynchttp.go:292: Completed job 16158705cbb9d71775d248f7f9c8508b in 10.365989731s >[negroni] Started GET /queue/16158705cbb9d71775d248f7f9c8508b >[negroni] Completed 303 See Other in 142.429µs >[negroni] Started GET /blockvolumes/192df25125cd0f660bd360b8c8b20d14 >[negroni] Completed 200 OK in 2.971282ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:54:30 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 22.756891ms >[asynchttp] INFO 2018/08/19 13:54:30 asynchttp.go:288: Started job e4e198a1788de3ecf4a674b20675c059 >[heketi] INFO 2018/08/19 13:54:30 Started async operation: Create Block Volume >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 113.193µs >[cmdexec] INFO 2018/08/19 13:54:30 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 51min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:54:31 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:54:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:54:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 129.972µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:54:32 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 24.019062ms >[asynchttp] INFO 2018/08/19 13:54:32 asynchttp.go:288: Started job 44b64923f80db745fc49577e8d989ba6 >[heketi] INFO 2018/08/19 13:54:32 Started async operation: Create Block Volume >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 106.356µs >[cmdexec] INFO 2018/08/19 13:54:32 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 51min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:54:32 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:54:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 126.131µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 128.518µs >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 150.651µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 129.379µs >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 193.025µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 130.472µs >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 200 OK in 138.597µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 125.646µs >[kubeexec] DEBUG 2018/08/19 13:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c20e473b943d491c2f8c7d53abc807f6 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:1fa20ce8-78d1-471b-8dc3-1ba53ac86db0", "USERNAME": "1fa20ce8-78d1-471b-8dc3-1ba53ac86db0", "PASSWORD": "8e51aba9-e74e-4ab2-86a6-a6adc99ccf7c", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:54:36 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:54:36 asynchttp.go:292: Completed job e4e198a1788de3ecf4a674b20675c059 in 5.606217235s >[kubeexec] DEBUG 2018/08/19 13:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:54:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e4e198a1788de3ecf4a674b20675c059 >[negroni] Completed 303 See Other in 165.838µs >[negroni] Started GET /blockvolumes/c20e473b943d491c2f8c7d53abc807f6 >[negroni] Completed 200 OK in 1.569123ms >[heketi] INFO 2018/08/19 13:54:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:54:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 169.765µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 259.131µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 155.96µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 126.386µs >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 200 OK in 139.571µs >[kubeexec] DEBUG 2018/08/19 13:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5ba9e7765c76c05ebbe16c4d03cac6da ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:08023a3f-d56e-4720-a320-ce6d0ba0337f", "USERNAME": "08023a3f-d56e-4720-a320-ce6d0ba0337f", "PASSWORD": "72ca4578-97c8-4d5d-b242-d593738a3154", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:54:41 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:54:41 asynchttp.go:292: Completed job 44b64923f80db745fc49577e8d989ba6 in 9.744224499s >[kubeexec] DEBUG 2018/08/19 13:54:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:54:41 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:54:41 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:54:42 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:54:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/44b64923f80db745fc49577e8d989ba6 >[negroni] Completed 303 See Other in 122.442µs >[negroni] Started GET /blockvolumes/5ba9e7765c76c05ebbe16c4d03cac6da >[negroni] Completed 200 OK in 2.474145ms >[kubeexec] DEBUG 2018/08/19 13:54:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 52min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:54:42 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:54:42 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:55:01 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.728257ms >[asynchttp] INFO 2018/08/19 13:55:01 asynchttp.go:288: Started job ceecb7a702cee2b4a080cf0ffa32f126 >[heketi] INFO 2018/08/19 13:55:01 Started async operation: Create Block Volume >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 109.927µs >[cmdexec] INFO 2018/08/19 13:55:01 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 52min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:55:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:55:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:55:01 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 121.777µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:55:02 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 39.38784ms >[asynchttp] INFO 2018/08/19 13:55:02 asynchttp.go:288: Started job aabbef1c1bc20f703d5ef673bbbb4892 >[heketi] INFO 2018/08/19 13:55:02 Started async operation: Create Block Volume >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 140.753µs >[cmdexec] INFO 2018/08/19 13:55:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 174.322µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 156.817µs >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 168.409µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 200.854µs >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 141.1µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 120.067µs >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 200 OK in 119.713µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 141.078µs >[kubeexec] DEBUG 2018/08/19 13:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_b0d86d379daf78a1107a05d50b71b232 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fe63db43-e048-4310-81d7-cacb6855d105", "USERNAME": "fe63db43-e048-4310-81d7-cacb6855d105", "PASSWORD": "b3ede3b7-60a7-4e48-95b3-8762a43f98fc", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:55:07 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:55:07 asynchttp.go:292: Completed job ceecb7a702cee2b4a080cf0ffa32f126 in 5.841824928s >[kubeexec] DEBUG 2018/08/19 13:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:55:07 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ceecb7a702cee2b4a080cf0ffa32f126 >[negroni] Completed 303 See Other in 147.232µs >[negroni] Started GET /blockvolumes/b0d86d379daf78a1107a05d50b71b232 >[negroni] Completed 200 OK in 2.114177ms >[kubeexec] DEBUG 2018/08/19 13:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 58min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:55:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 130.699µs >[kubeexec] DEBUG 2018/08/19 13:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 52min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:55:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 58min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 134.752µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 118.61µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 120.793µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 128.225µs >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 200 OK in 172.218µs >[kubeexec] DEBUG 2018/08/19 13:55:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d8751eac8fc0ed3ea961be46aa26dd17 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:10b75bb0-ec3c-4dda-a789-c8dc42cae71e", "USERNAME": "10b75bb0-ec3c-4dda-a789-c8dc42cae71e", "PASSWORD": "23406aee-e8f6-4334-951b-2b952880d4c8", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:55:12 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:55:12 asynchttp.go:292: Completed job aabbef1c1bc20f703d5ef673bbbb4892 in 10.183092596s >[negroni] Started GET /queue/aabbef1c1bc20f703d5ef673bbbb4892 >[negroni] Completed 303 See Other in 217.398µs >[negroni] Started GET /blockvolumes/d8751eac8fc0ed3ea961be46aa26dd17 >[negroni] Completed 200 OK in 3.438893ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:55:31 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 31.954122ms >[asynchttp] INFO 2018/08/19 13:55:31 asynchttp.go:288: Started job 0b0fca3ab69940e0a1aecf2c16cfebb0 >[heketi] INFO 2018/08/19 13:55:31 Started async operation: Create Block Volume >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 98.252µs >[cmdexec] INFO 2018/08/19 13:55:31 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 59min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:55:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:55:31 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:31 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 52min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:55:31 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 135.308µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:55:32 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 53.391333ms >[asynchttp] INFO 2018/08/19 13:55:32 asynchttp.go:288: Started job 355472a1cd254471082a3b0eb76e8543 >[heketi] INFO 2018/08/19 13:55:32 Started async operation: Create Block Volume >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 190.826µs >[cmdexec] INFO 2018/08/19 13:55:32 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 52min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:55:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 144.153µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 199.051µs >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 121.47µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 171.18µs >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 173.605µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 125.749µs >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 200 OK in 173.113µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 122.734µs >[kubeexec] DEBUG 2018/08/19 13:55:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d306c6b4b71ad797e9d94a2c16b51dba ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a602da9e-a08c-490e-a2f5-7bd1c95b14f6", "USERNAME": "a602da9e-a08c-490e-a2f5-7bd1c95b14f6", "PASSWORD": "5c47b2d1-23c0-45dc-a3c9-3469a6c1ee4a", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:55:37 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:55:37 asynchttp.go:292: Completed job 0b0fca3ab69940e0a1aecf2c16cfebb0 in 5.611984652s >[kubeexec] DEBUG 2018/08/19 13:55:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:55:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 59min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:55:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:55:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0b0fca3ab69940e0a1aecf2c16cfebb0 >[negroni] Completed 303 See Other in 169.279µs >[negroni] Started GET /blockvolumes/d306c6b4b71ad797e9d94a2c16b51dba >[negroni] Completed 200 OK in 1.399315ms >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 174.433µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 158.544µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 144.953µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 139.048µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 313.477µs >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 200 OK in 196.074µs >[kubeexec] DEBUG 2018/08/19 13:55:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_00b7b07b794c857f379d52961d397fdf ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:cebfa46d-156e-412c-a283-4d5b74bc4de6", "USERNAME": "cebfa46d-156e-412c-a283-4d5b74bc4de6", "PASSWORD": "632cf67e-1bd0-4179-83b3-01c4f570717d", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:55:42 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:55:42 asynchttp.go:292: Completed job 355472a1cd254471082a3b0eb76e8543 in 10.06357821s >[negroni] Started GET /queue/355472a1cd254471082a3b0eb76e8543 >[negroni] Completed 303 See Other in 186.868µs >[negroni] Started GET /blockvolumes/00b7b07b794c857f379d52961d397fdf >[negroni] Completed 200 OK in 3.663603ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:56:01 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.679358ms >[asynchttp] INFO 2018/08/19 13:56:01 asynchttp.go:288: Started job dbdb96535c4d84af495d5eff73c49e74 >[heketi] INFO 2018/08/19 13:56:01 Started async operation: Create Block Volume >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 88.214µs >[cmdexec] INFO 2018/08/19 13:56:01 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:01 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 53min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:56:01 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 59min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:56:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:56:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 182.216µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:56:03 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 36.310249ms >[asynchttp] INFO 2018/08/19 13:56:03 asynchttp.go:288: Started job eec987d3b476bce2e4b25c4ec8b8fb16 >[heketi] INFO 2018/08/19 13:56:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 99.275µs >[cmdexec] INFO 2018/08/19 13:56:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 1h 59min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:56:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 162.013µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 135.96µs >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 125.723µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 163.894µs >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 173.506µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 187.716µs >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 200 OK in 139.061µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 120.718µs >[kubeexec] DEBUG 2018/08/19 13:56:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_3ad7cede111ea7c2e344ca91741f82cb ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3f78b902-3602-4f3e-8985-01896e97173f", "USERNAME": "3f78b902-3602-4f3e-8985-01896e97173f", "PASSWORD": "90c2ed0d-6ec5-4ed5-951a-e020782193ad", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:56:07 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:56:07 asynchttp.go:292: Completed job dbdb96535c4d84af495d5eff73c49e74 in 5.798152613s >[kubeexec] DEBUG 2018/08/19 13:56:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/dbdb96535c4d84af495d5eff73c49e74 >[negroni] Completed 303 See Other in 152.12µs >[negroni] Started GET /blockvolumes/3ad7cede111ea7c2e344ca91741f82cb >[negroni] Completed 200 OK in 2.363362ms >[cmdexec] INFO 2018/08/19 13:56:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 53min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:56:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 177.305µs >[kubeexec] DEBUG 2018/08/19 13:56:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 1h 59min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 131.321µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 199.491µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 181.96µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 151.195µs >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 200 OK in 132.723µs >[kubeexec] DEBUG 2018/08/19 13:56:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d618d29cd43384616dff9f24f2324e5a ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:82cbc836-df19-4987-8919-ff01a10fc813", "USERNAME": "82cbc836-df19-4987-8919-ff01a10fc813", "PASSWORD": "7b948e9d-66ac-4b68-80be-47516fa0ecd9", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:56:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:56:13 asynchttp.go:292: Completed job eec987d3b476bce2e4b25c4ec8b8fb16 in 10.637196235s >[negroni] Started GET /queue/eec987d3b476bce2e4b25c4ec8b8fb16 >[negroni] Completed 303 See Other in 137.873µs >[negroni] Started GET /blockvolumes/d618d29cd43384616dff9f24f2324e5a >[negroni] Completed 200 OK in 2.898356ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:56:32 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 28.679075ms >[asynchttp] INFO 2018/08/19 13:56:32 asynchttp.go:288: Started job 9cfa75b80c50a11791f912f2cd8052a5 >[heketi] INFO 2018/08/19 13:56:32 Started async operation: Create Block Volume >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 126.009µs >[cmdexec] INFO 2018/08/19 13:56:32 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 53min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:56:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:56:32 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:56:32 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:32 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 132.789µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:56:33 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.495614ms >[asynchttp] INFO 2018/08/19 13:56:33 asynchttp.go:288: Started job 9192446c40ef0c9ec1e6fdcbb0c60905 >[heketi] INFO 2018/08/19 13:56:33 Started async operation: Create Block Volume >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 105.225µs >[cmdexec] INFO 2018/08/19 13:56:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 123.949µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 161.251µs >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 164.804µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 154.297µs >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 135.378µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 123.029µs >[heketi] INFO 2018/08/19 13:56:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:56:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 200 OK in 122.518µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 162.604µs >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d3e7439be0a0cd98f83a2eb3bf438aff ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:feceb8bb-812d-4174-808c-0b88297de476", "USERNAME": "feceb8bb-812d-4174-808c-0b88297de476", "PASSWORD": "835d1228-1ecd-47de-b07c-159d38ae0e5f", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:56:38 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:56:38 asynchttp.go:292: Completed job 9cfa75b80c50a11791f912f2cd8052a5 in 5.819481909s >[negroni] Started GET /queue/9cfa75b80c50a11791f912f2cd8052a5 >[negroni] Completed 303 See Other in 190.493µs >[negroni] Started GET /blockvolumes/d3e7439be0a0cd98f83a2eb3bf438aff >[negroni] Completed 200 OK in 3.485694ms >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:56:38 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 132.191µs >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:56:38 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:56:38 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:56:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:56:38 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:56:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 54min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:56:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:56:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] DEBUG 2018/08/19 13:56:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 54min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:56:39 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:56:39 Cleaned 0 nodes from health cache >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 176.311µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 121.311µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 146.483µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 137.497µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 130.62µs >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 200 OK in 130.915µs >[kubeexec] DEBUG 2018/08/19 13:56:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_85e65fa975ac43c9008d486b3b2a3278 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f677d080-ce4d-4780-b977-da8cf551a6fb", "USERNAME": "f677d080-ce4d-4780-b977-da8cf551a6fb", "PASSWORD": "b081249b-570a-4b75-b9b3-8e18e05184fa", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:56:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:56:44 asynchttp.go:292: Completed job 9192446c40ef0c9ec1e6fdcbb0c60905 in 11.188428363s >[negroni] Started GET /queue/9192446c40ef0c9ec1e6fdcbb0c60905 >[negroni] Completed 303 See Other in 181.215µs >[negroni] Started GET /blockvolumes/85e65fa975ac43c9008d486b3b2a3278 >[negroni] Completed 200 OK in 3.515212ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:57:02 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 27.927813ms >[asynchttp] INFO 2018/08/19 13:57:02 asynchttp.go:288: Started job 029d5ef92a5d79ba89f76d18006f77b1 >[heketi] INFO 2018/08/19 13:57:02 Started async operation: Create Block Volume >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 106.918µs >[cmdexec] INFO 2018/08/19 13:57:02 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:57:02 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:02 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 54min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:57:02 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:57:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 171.221µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:57:03 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 35.466013ms >[asynchttp] INFO 2018/08/19 13:57:03 asynchttp.go:288: Started job bea56a7f7e2993eb538aae82120baa1c >[heketi] INFO 2018/08/19 13:57:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 114.108µs >[cmdexec] INFO 2018/08/19 13:57:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 144.16µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 157.682µs >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 158.07µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 126.141µs >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 126.112µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 118.955µs >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 148.267µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 178.675µs >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 200 OK in 127.107µs >[kubeexec] DEBUG 2018/08/19 13:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f16ffbd19d77d74587435ee34dfea7aa ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:09589377-70f4-4178-aa08-c33904915308", "USERNAME": "09589377-70f4-4178-aa08-c33904915308", "PASSWORD": "3798aa9c-ccbd-4867-972d-db22ff88a88a", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:57:08 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:57:08 asynchttp.go:292: Completed job 029d5ef92a5d79ba89f76d18006f77b1 in 6.051545388s >[kubeexec] DEBUG 2018/08/19 13:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:57:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 105.814µs >[kubeexec] DEBUG 2018/08/19 13:57:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 54min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:57:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 0min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:57:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 0min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/029d5ef92a5d79ba89f76d18006f77b1 >[negroni] Completed 303 See Other in 119.973µs >[negroni] Started GET /blockvolumes/f16ffbd19d77d74587435ee34dfea7aa >[negroni] Completed 200 OK in 1.4479ms >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 132.519µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 164.434µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 185.932µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 141.714µs >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 200 OK in 133.808µs >[kubeexec] DEBUG 2018/08/19 13:57:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_911eb5ef0de0cdbf57f6645037a3ccc7 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6d4663db-c0de-468e-ae5d-1338cc0b554f", "USERNAME": "6d4663db-c0de-468e-ae5d-1338cc0b554f", "PASSWORD": "4b152649-f74d-455d-9c23-d851641f12ea", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:57:14 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:57:14 asynchttp.go:292: Completed job bea56a7f7e2993eb538aae82120baa1c in 10.743864421s >[negroni] Started GET /queue/bea56a7f7e2993eb538aae82120baa1c >[negroni] Completed 303 See Other in 163.206µs >[negroni] Started GET /blockvolumes/911eb5ef0de0cdbf57f6645037a3ccc7 >[negroni] Completed 200 OK in 2.63725ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:57:32 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 103.110231ms >[asynchttp] INFO 2018/08/19 13:57:33 asynchttp.go:288: Started job da8e6efb98f154512a47259b38b06e2e >[heketi] INFO 2018/08/19 13:57:33 Started async operation: Create Block Volume >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 102.603µs >[cmdexec] INFO 2018/08/19 13:57:33 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 1min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:57:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:57:33 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 54min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:57:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 153.374µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:57:34 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 26.727497ms >[asynchttp] INFO 2018/08/19 13:57:34 asynchttp.go:288: Started job f310f9079009f78eeb06e4103a507dfc >[heketi] INFO 2018/08/19 13:57:34 Started async operation: Create Block Volume >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 138.783µs >[cmdexec] INFO 2018/08/19 13:57:34 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 55min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:57:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 164.655µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 124.726µs >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 186.505µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 149.276µs >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 125.663µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 125.569µs >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 200 OK in 169.654µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 180.51µs >[kubeexec] DEBUG 2018/08/19 13:57:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_81bcc0d915e0130a3ab9ea0e62c23b2b ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:099a7f95-c2b9-49d8-9e49-077c5938a46c", "USERNAME": "099a7f95-c2b9-49d8-9e49-077c5938a46c", "PASSWORD": "c86f4130-2899-45fb-99f7-5bad10a57082", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:57:38 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:57:38 asynchttp.go:292: Completed job da8e6efb98f154512a47259b38b06e2e in 5.874351186s >[kubeexec] DEBUG 2018/08/19 13:57:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:57:39 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/da8e6efb98f154512a47259b38b06e2e >[negroni] Completed 303 See Other in 129.827µs >[negroni] Started GET /blockvolumes/81bcc0d915e0130a3ab9ea0e62c23b2b >[negroni] Completed 200 OK in 2.136227ms >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 153.104µs >[kubeexec] DEBUG 2018/08/19 13:57:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 1min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:57:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:57:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 130.972µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 183.734µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 127.518µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 165.3µs >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 200 OK in 160.592µs >[kubeexec] DEBUG 2018/08/19 13:57:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5b21c52a2ac6a1e083741f78b1498123 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b608a230-20f4-4fe4-9a0e-5572288f5c46", "USERNAME": "b608a230-20f4-4fe4-9a0e-5572288f5c46", "PASSWORD": "3ff64081-1d0e-4a62-a262-6c45c751448d", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:57:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:57:44 asynchttp.go:292: Completed job f310f9079009f78eeb06e4103a507dfc in 10.817196002s >[negroni] Started GET /queue/f310f9079009f78eeb06e4103a507dfc >[negroni] Completed 303 See Other in 155.408µs >[negroni] Started GET /blockvolumes/5b21c52a2ac6a1e083741f78b1498123 >[negroni] Completed 200 OK in 2.700382ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:58:03 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 76.110967ms >[asynchttp] INFO 2018/08/19 13:58:03 asynchttp.go:288: Started job 787d4dbf2e173b88882ca91cc840cf19 >[heketi] INFO 2018/08/19 13:58:03 Started async operation: Create Block Volume >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 138.421µs >[cmdexec] INFO 2018/08/19 13:58:03 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 55min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:58:03 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 1min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:58:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:03 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:58:03 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 200.849µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:58:04 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 31.188013ms >[asynchttp] INFO 2018/08/19 13:58:04 asynchttp.go:288: Started job 3b866c7dbea08b1897a5ac4432412438 >[heketi] INFO 2018/08/19 13:58:04 Started async operation: Create Block Volume >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 131.662µs >[cmdexec] INFO 2018/08/19 13:58:04 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 1min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:58:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 172.28µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 146.199µs >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 141.153µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 130.041µs >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 132.556µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 133.356µs >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 200 OK in 125.162µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 161.342µs >[kubeexec] DEBUG 2018/08/19 13:58:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_9b369489faad3eda12b646e4194466cb ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:cae5c162-7278-4862-9e24-1eaed8828d0c", "USERNAME": "cae5c162-7278-4862-9e24-1eaed8828d0c", "PASSWORD": "72f27342-9583-4f29-923b-f3f3e9ed2efe", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:58:09 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:58:09 asynchttp.go:292: Completed job 787d4dbf2e173b88882ca91cc840cf19 in 5.978158001s >[negroni] Started GET /queue/787d4dbf2e173b88882ca91cc840cf19 >[negroni] Completed 303 See Other in 132.34µs >[negroni] Started GET /blockvolumes/9b369489faad3eda12b646e4194466cb >[negroni] Completed 200 OK in 2.813473ms >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 270.606µs >[kubeexec] DEBUG 2018/08/19 13:58:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:58:09 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 55min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:58:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 1min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 145.374µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 132.13µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 140.27µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 128.102µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 677.291µs >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 200 OK in 167.693µs >[kubeexec] DEBUG 2018/08/19 13:58:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c624549a35cdb53e261d180bbd73fe15 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:21f3703c-e2ce-41b8-807a-ba8601a01c41", "USERNAME": "21f3703c-e2ce-41b8-807a-ba8601a01c41", "PASSWORD": "391fc72f-e900-4946-84b5-bde9e9cba864", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:58:15 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:58:15 asynchttp.go:292: Completed job 3b866c7dbea08b1897a5ac4432412438 in 11.032334494s >[negroni] Started GET /queue/3b866c7dbea08b1897a5ac4432412438 >[negroni] Completed 303 See Other in 199.441µs >[negroni] Started GET /blockvolumes/c624549a35cdb53e261d180bbd73fe15 >[negroni] Completed 200 OK in 2.538396ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:58:33 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 31.820145ms >[asynchttp] INFO 2018/08/19 13:58:33 asynchttp.go:288: Started job d92e6b326d90706081685e5fb0895f42 >[heketi] INFO 2018/08/19 13:58:33 Started async operation: Create Block Volume >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 145.559µs >[cmdexec] INFO 2018/08/19 13:58:33 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:33 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:58:33 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:58:34 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 56min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:58:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 129.263µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:58:34 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 24.001007ms >[asynchttp] INFO 2018/08/19 13:58:34 asynchttp.go:288: Started job 57993ae803f0b9c017f7ea21454a9bd1 >[heketi] INFO 2018/08/19 13:58:34 Started async operation: Create Block Volume >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 118.045µs >[cmdexec] INFO 2018/08/19 13:58:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 123.933µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 123.555µs >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 129.031µs >[heketi] INFO 2018/08/19 13:58:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 13:58:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 122.865µs >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 127.585µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 172.716µs >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 128.551µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 173.592µs >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 200 OK in 136.81µs >[kubeexec] DEBUG 2018/08/19 13:58:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_60fd96715f97abd9e6060148835279f5 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:9f2db708-be67-4bcc-8b28-fb7f13b7e1d8", "USERNAME": "9f2db708-be67-4bcc-8b28-fb7f13b7e1d8", "PASSWORD": "71bd47db-48bc-47ec-be2d-9f42124fdcf4", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 104.877µs >[heketi] INFO 2018/08/19 13:58:40 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:58:40 asynchttp.go:292: Completed job d92e6b326d90706081685e5fb0895f42 in 6.18486068s >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:58:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 13:58:40 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 13:58:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 56min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:58:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 13:58:40 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 13:58:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 56min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 13:58:40 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 13:58:40 Cleaned 0 nodes from health cache >[cmdexec] INFO 2018/08/19 13:58:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:58:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/d92e6b326d90706081685e5fb0895f42 >[negroni] Completed 303 See Other in 136.655µs >[negroni] Started GET /blockvolumes/60fd96715f97abd9e6060148835279f5 >[negroni] Completed 200 OK in 1.355025ms >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 134.028µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 114.157µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 120.64µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 144.835µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 186.247µs >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 200 OK in 159.477µs >[kubeexec] DEBUG 2018/08/19 13:58:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f6fd70be219b2ac14f904e96feb5937f ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:28dff8d0-c187-467e-8717-4a0f97b83927", "USERNAME": "28dff8d0-c187-467e-8717-4a0f97b83927", "PASSWORD": "cca66f1b-438a-4fa3-ac37-1306febd8d92", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:58:46 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:58:46 asynchttp.go:292: Completed job 57993ae803f0b9c017f7ea21454a9bd1 in 11.628620132s >[negroni] Started GET /queue/57993ae803f0b9c017f7ea21454a9bd1 >[negroni] Completed 303 See Other in 146.932µs >[negroni] Started GET /blockvolumes/f6fd70be219b2ac14f904e96feb5937f >[negroni] Completed 200 OK in 2.622906ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:59:04 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 31.964135ms >[asynchttp] INFO 2018/08/19 13:59:04 asynchttp.go:288: Started job 941d0d1900b9949f21db04a6ab169c4c >[heketi] INFO 2018/08/19 13:59:04 Started async operation: Create Block Volume >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 104.845µs >[cmdexec] INFO 2018/08/19 13:59:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:59:04 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 56min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:59:04 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:59:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:04 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 2min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 128.6µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:59:05 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 77.480981ms >[asynchttp] INFO 2018/08/19 13:59:05 asynchttp.go:288: Started job c741231306f340b7462dcb76e4f3a568 >[heketi] INFO 2018/08/19 13:59:05 Started async operation: Create Block Volume >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 100.573µs >[cmdexec] INFO 2018/08/19 13:59:05 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 2min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:59:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 177.831µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 211.456µs >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 163.221µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 164.499µs >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 213.223µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 125.48µs >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 123.305µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 129.731µs >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 200 OK in 169.048µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 169.485µs >[kubeexec] DEBUG 2018/08/19 13:59:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fe4c080c821c0452e87c692b60f83757 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e90d4b8d-38b2-4bfa-bbfd-96ef55315741", "USERNAME": "e90d4b8d-38b2-4bfa-bbfd-96ef55315741", "PASSWORD": "d7516421-6f03-46db-b67a-e733a71a8ecd", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:59:10 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:59:10 asynchttp.go:292: Completed job 941d0d1900b9949f21db04a6ab169c4c in 6.337846456s >[kubeexec] DEBUG 2018/08/19 13:59:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:59:10 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 56min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:59:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/941d0d1900b9949f21db04a6ab169c4c >[negroni] Completed 303 See Other in 214.127µs >[negroni] Started GET /blockvolumes/fe4c080c821c0452e87c692b60f83757 >[negroni] Completed 200 OK in 1.816293ms >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 184.571µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 123.241µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 171.203µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 135.66µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 156.145µs >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 200 OK in 124.038µs >[kubeexec] DEBUG 2018/08/19 13:59:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0b7df76ae071da269c7f0e88af0695cd ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:da59c40c-6ced-4358-bd5b-b119b6c73533", "USERNAME": "da59c40c-6ced-4358-bd5b-b119b6c73533", "PASSWORD": "813eccd9-ecea-49c2-b91a-e5350a051b9d", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:59:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:59:16 asynchttp.go:292: Completed job c741231306f340b7462dcb76e4f3a568 in 11.348765788s >[negroni] Started GET /queue/c741231306f340b7462dcb76e4f3a568 >[negroni] Completed 303 See Other in 203.074µs >[negroni] Started GET /blockvolumes/0b7df76ae071da269c7f0e88af0695cd >[negroni] Completed 200 OK in 2.711755ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:59:34 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 87.348692ms >[asynchttp] INFO 2018/08/19 13:59:34 asynchttp.go:288: Started job 2f630f2b7e83e96f04c79c298c3c428b >[heketi] INFO 2018/08/19 13:59:34 Started async operation: Create Block Volume >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 125.712µs >[cmdexec] INFO 2018/08/19 13:59:34 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:59:34 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:34 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 3min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:59:34 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 57min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:59:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 127.921µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 13:59:35 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 28.00546ms >[asynchttp] INFO 2018/08/19 13:59:35 asynchttp.go:288: Started job 535c9884edccc970b41770625947103a >[heketi] INFO 2018/08/19 13:59:35 Started async operation: Create Block Volume >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 89.996µs >[cmdexec] INFO 2018/08/19 13:59:35 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 3min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 13:59:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 130.087µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 141.014µs >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 150.342µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 116.478µs >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 124.129µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 132.277µs >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 148.566µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 130.162µs >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 200 OK in 129.156µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 228.244µs >[kubeexec] DEBUG 2018/08/19 13:59:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c4618852a445466e223b086c37f8c3c4 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:373a9815-f755-4058-8928-f830280b5a09", "USERNAME": "373a9815-f755-4058-8928-f830280b5a09", "PASSWORD": "6de21796-3b19-4414-a0af-201b355311c7", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:59:40 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:59:40 asynchttp.go:292: Completed job 2f630f2b7e83e96f04c79c298c3c428b in 6.26534193s >[kubeexec] DEBUG 2018/08/19 13:59:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 13:59:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 57min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 13:59:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 13:59:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2f630f2b7e83e96f04c79c298c3c428b >[negroni] Completed 303 See Other in 156.412µs >[negroni] Started GET /blockvolumes/c4618852a445466e223b086c37f8c3c4 >[negroni] Completed 200 OK in 1.552331ms >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 217.917µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 140.635µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 121.139µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 127.213µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 178.386µs >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 200 OK in 121.125µs >[kubeexec] DEBUG 2018/08/19 13:59:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_46415562268cad60c99ce99e2f88b8b2 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7be5eb7e-797e-42d5-af13-440d3a11ff29", "USERNAME": "7be5eb7e-797e-42d5-af13-440d3a11ff29", "PASSWORD": "140441dd-d585-416d-a9d3-72602875ea8a", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 13:59:47 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 13:59:47 asynchttp.go:292: Completed job 535c9884edccc970b41770625947103a in 11.343082345s >[negroni] Started GET /queue/535c9884edccc970b41770625947103a >[negroni] Completed 303 See Other in 164.284µs >[negroni] Started GET /blockvolumes/46415562268cad60c99ce99e2f88b8b2 >[negroni] Completed 200 OK in 2.508155ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:00:04 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 29.429551ms >[asynchttp] INFO 2018/08/19 14:00:04 asynchttp.go:288: Started job 19ea2cbcb03856baa4f264b0206566eb >[heketi] INFO 2018/08/19 14:00:04 Started async operation: Create Block Volume >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 160.823µs >[cmdexec] INFO 2018/08/19 14:00:04 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:00:05 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 3min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:00:05 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 57min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:00:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 3min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 180.615µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:00:06 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 29.336755ms >[asynchttp] INFO 2018/08/19 14:00:06 asynchttp.go:288: Started job 7cc64059688e53a96758c6bd3e942ead >[heketi] INFO 2018/08/19 14:00:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 100.84µs >[cmdexec] INFO 2018/08/19 14:00:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 57min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:00:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 182.119µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 271.973µs >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 138.44µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 166.951µs >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 196.345µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 134.357µs >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 138.15µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 162.945µs >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 200 OK in 142.071µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 134.725µs >[kubeexec] DEBUG 2018/08/19 14:00:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_beec30260a1611f6dadd2e7bb7d41818 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c33d584b-ba4d-4c50-9edd-2fc6bb3231b5", "USERNAME": "c33d584b-ba4d-4c50-9edd-2fc6bb3231b5", "PASSWORD": "1ce1942b-a048-4012-9d36-a897c95a043b", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:00:11 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:00:11 asynchttp.go:292: Completed job 19ea2cbcb03856baa4f264b0206566eb in 6.244509856s >[kubeexec] DEBUG 2018/08/19 14:00:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:00:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 3min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:00:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/19ea2cbcb03856baa4f264b0206566eb >[negroni] Completed 303 See Other in 175.787µs >[negroni] Started GET /blockvolumes/beec30260a1611f6dadd2e7bb7d41818 >[negroni] Completed 200 OK in 1.767637ms >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 216.253µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 134.616µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 142.661µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 166.853µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 122.802µs >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 200 OK in 142.022µs >[kubeexec] DEBUG 2018/08/19 14:00:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ae29c06cd140ecf1477b6537fb95baf4 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:183e6ca0-5cbb-4fd4-8a06-c615bd3b0591", "USERNAME": "183e6ca0-5cbb-4fd4-8a06-c615bd3b0591", "PASSWORD": "023fc1fa-db1d-49c8-be5c-88500f88d279", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:00:17 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:00:17 asynchttp.go:292: Completed job 7cc64059688e53a96758c6bd3e942ead in 11.236364147s >[negroni] Started GET /queue/7cc64059688e53a96758c6bd3e942ead >[negroni] Completed 303 See Other in 142.031µs >[negroni] Started GET /blockvolumes/ae29c06cd140ecf1477b6537fb95baf4 >[negroni] Completed 200 OK in 2.727596ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:00:35 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 33.053271ms >[asynchttp] INFO 2018/08/19 14:00:35 asynchttp.go:288: Started job 429076506efadaf91e6fdff314c6f08b >[heketi] INFO 2018/08/19 14:00:35 Started async operation: Create Block Volume >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 113.199µs >[cmdexec] INFO 2018/08/19 14:00:35 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 58min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:00:35 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:00:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:00:35 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:35 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 134.89µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:00:36 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 28.991851ms >[asynchttp] INFO 2018/08/19 14:00:36 asynchttp.go:288: Started job 9a472d7429690eb4d61a79bf54c89696 >[heketi] INFO 2018/08/19 14:00:36 Started async operation: Create Block Volume >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 105.31µs >[cmdexec] INFO 2018/08/19 14:00:36 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 58min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:00:36 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:00:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 14:00:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:00:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 207.176µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 138.513µs >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 126.939µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 126.078µs >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 118.875µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 152.985µs >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 183.827µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 147.723µs >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 200 OK in 145.3µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 128.357µs >[kubeexec] DEBUG 2018/08/19 14:00:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_eca68355eef4c939d53f290633c60f1c ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:7f6c8608-9693-4439-9434-9d3a4ca32c71", "USERNAME": "7f6c8608-9693-4439-9434-9d3a4ca32c71", "PASSWORD": "44734cc3-0bb6-45fb-bb16-9b721fb46b0a", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:00:41 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:00:41 asynchttp.go:292: Completed job 429076506efadaf91e6fdff314c6f08b in 6.360071948s >[kubeexec] DEBUG 2018/08/19 14:00:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:00:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:00:41 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:00:41 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] DEBUG 2018/08/19 14:00:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:00:42 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:00:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:00:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 58min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:00:42 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:00:42 Cleaned 0 nodes from health cache >[negroni] Started GET /queue/429076506efadaf91e6fdff314c6f08b >[negroni] Completed 303 See Other in 218.386µs >[negroni] Started GET /blockvolumes/eca68355eef4c939d53f290633c60f1c >[negroni] Completed 200 OK in 1.403303ms >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 174.02µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 174.748µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 189.061µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 174.929µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 184.284µs >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 200 OK in 183.646µs >[kubeexec] DEBUG 2018/08/19 14:00:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ac0edbc6cce9b695ea056cb7f21f292a ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:466de314-f538-494d-8fec-91de167a090f", "USERNAME": "466de314-f538-494d-8fec-91de167a090f", "PASSWORD": "95ad6dad-565f-4e8d-8d8d-9f31e1bfec5c", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:00:47 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:00:47 asynchttp.go:292: Completed job 9a472d7429690eb4d61a79bf54c89696 in 11.511944284s >[negroni] Started GET /queue/9a472d7429690eb4d61a79bf54c89696 >[negroni] Completed 303 See Other in 199.511µs >[negroni] Started GET /blockvolumes/ac0edbc6cce9b695ea056cb7f21f292a >[negroni] Completed 200 OK in 3.087533ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:01:05 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 31.768846ms >[asynchttp] INFO 2018/08/19 14:01:05 asynchttp.go:288: Started job 787db1cd073d859f76b75f717354c2bc >[heketi] INFO 2018/08/19 14:01:05 Started async operation: Create Block Volume >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 103.948µs >[cmdexec] INFO 2018/08/19 14:01:05 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 58min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:01:05 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:05 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:01:05 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:01:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 4min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 144.838µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:01:06 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 32.38062ms >[asynchttp] INFO 2018/08/19 14:01:06 asynchttp.go:288: Started job 2d5a17002cb724ee4bbbaad5525a573a >[heketi] INFO 2018/08/19 14:01:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 113.774µs >[cmdexec] INFO 2018/08/19 14:01:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 58min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:01:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 136.79µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 126.54µs >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 128.231µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 126.741µs >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 170.188µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 126.41µs >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 118.743µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 226.152µs >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 200 OK in 167.426µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 120.943µs >[kubeexec] DEBUG 2018/08/19 14:01:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ba716a4891174cdf460786dbbdd78d12 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:961d1246-1d61-4ecc-89ca-ed86d7b80247", "USERNAME": "961d1246-1d61-4ecc-89ca-ed86d7b80247", "PASSWORD": "2acf7877-e98a-4d14-99e2-c1698f1784ce", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:01:12 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:01:12 asynchttp.go:292: Completed job 787db1cd073d859f76b75f717354c2bc in 6.680424509s >[kubeexec] DEBUG 2018/08/19 14:01:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:01:12 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 4min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:01:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/787db1cd073d859f76b75f717354c2bc >[negroni] Completed 303 See Other in 144.416µs >[negroni] Started GET /blockvolumes/ba716a4891174cdf460786dbbdd78d12 >[negroni] Completed 200 OK in 1.528084ms >[kubeexec] DEBUG 2018/08/19 14:01:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 126.301µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 132.603µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 140.316µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 179.317µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 151.544µs >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 200 OK in 128.891µs >[kubeexec] DEBUG 2018/08/19 14:01:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_a5bf4b1fb6ab89780231fe66c023eddb ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:12d1e549-f6c3-4daf-b825-222dd42d3a5e", "USERNAME": "12d1e549-f6c3-4daf-b825-222dd42d3a5e", "PASSWORD": "0aa3b6fc-5216-4591-9b9a-9cffd4d853b1", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:01:18 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:01:18 asynchttp.go:292: Completed job 2d5a17002cb724ee4bbbaad5525a573a in 11.878520166s >[negroni] Started GET /queue/2d5a17002cb724ee4bbbaad5525a573a >[negroni] Completed 303 See Other in 150.638µs >[negroni] Started GET /blockvolumes/a5bf4b1fb6ab89780231fe66c023eddb >[negroni] Completed 200 OK in 2.702603ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:01:35 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 39.014754ms >[asynchttp] INFO 2018/08/19 14:01:36 asynchttp.go:288: Started job 9bd8744b67b32b8a65422cc468fdf1d8 >[heketi] INFO 2018/08/19 14:01:36 Started async operation: Create Block Volume >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 166.37µs >[cmdexec] INFO 2018/08/19 14:01:36 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 59min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:01:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:01:36 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 5min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:01:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 120.6µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:01:37 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 34.603958ms >[asynchttp] INFO 2018/08/19 14:01:37 asynchttp.go:288: Started job 693880226f2df5c73414ca4e17c6c7e9 >[heketi] INFO 2018/08/19 14:01:37 Started async operation: Create Block Volume >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 112.109µs >[cmdexec] INFO 2018/08/19 14:01:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 5min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:01:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 174.921µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 154.815µs >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 225.817µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 125.002µs >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 213.384µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 132.699µs >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 168.892µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 124.152µs >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 200 OK in 191.151µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 143.86µs >[kubeexec] DEBUG 2018/08/19 14:01:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_537474bcb2f867d5ebb88d184899a86d ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c8655c38-f844-4eab-aeda-f3b3332faca1", "USERNAME": "c8655c38-f844-4eab-aeda-f3b3332faca1", "PASSWORD": "518ab6f1-7b08-4943-b69d-e919c0724360", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:01:42 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:01:42 asynchttp.go:292: Completed job 9bd8744b67b32b8a65422cc468fdf1d8 in 6.75624587s >[kubeexec] DEBUG 2018/08/19 14:01:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:01:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:01:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 59min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[negroni] Started GET /queue/9bd8744b67b32b8a65422cc468fdf1d8 >[negroni] Completed 303 See Other in 113.597µs >[cmdexec] INFO 2018/08/19 14:01:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /blockvolumes/537474bcb2f867d5ebb88d184899a86d >[negroni] Completed 200 OK in 1.21715ms >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 126.178µs >[kubeexec] DEBUG 2018/08/19 14:01:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 131.767µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 125.607µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 120.631µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 123.454µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 177.147µs >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 200 OK in 151.189µs >[kubeexec] DEBUG 2018/08/19 14:01:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_61ba095d52a0abc496fd2e81f9fb5113 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a0db50a9-426c-4a07-bd4d-f3babf2a7291", "USERNAME": "a0db50a9-426c-4a07-bd4d-f3babf2a7291", "PASSWORD": "51a3f64f-dab7-40d4-a0fc-33bd08282c0c", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:01:49 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:01:49 asynchttp.go:292: Completed job 693880226f2df5c73414ca4e17c6c7e9 in 12.136516176s >[negroni] Started GET /queue/693880226f2df5c73414ca4e17c6c7e9 >[negroni] Completed 303 See Other in 200.553µs >[negroni] Started GET /blockvolumes/61ba095d52a0abc496fd2e81f9fb5113 >[negroni] Completed 200 OK in 3.682837ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:02:06 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 41.872056ms >[asynchttp] INFO 2018/08/19 14:02:06 asynchttp.go:288: Started job 974553f8fd06578532141df71629aa2b >[heketi] INFO 2018/08/19 14:02:06 Started async operation: Create Block Volume >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 142.679µs >[cmdexec] INFO 2018/08/19 14:02:06 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 59min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:02:06 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 5min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:02:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:02:06 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:06 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 5min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 131.287µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:02:07 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 89.365606ms >[asynchttp] INFO 2018/08/19 14:02:07 asynchttp.go:288: Started job 35c885227f8cdd05d212e46d7eb172fb >[heketi] INFO 2018/08/19 14:02:07 Started async operation: Create Block Volume >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 158.108µs >[cmdexec] INFO 2018/08/19 14:02:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 130.574µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 133.696µs >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 170.452µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 121.433µs >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 139.756µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 155.365µs >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 122.453µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 140.221µs >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 200 OK in 215.157µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 179.938µs >[kubeexec] DEBUG 2018/08/19 14:02:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_a596757f15215a4b45589ee382b023b2 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ef7935f7-749a-42a7-9fae-3b46d4432f5b", "USERNAME": "ef7935f7-749a-42a7-9fae-3b46d4432f5b", "PASSWORD": "e4baab12-b659-4e50-98c4-ec4165be6253", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:02:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:02:13 asynchttp.go:292: Completed job 974553f8fd06578532141df71629aa2b in 6.681176354s >[kubeexec] DEBUG 2018/08/19 14:02:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:02:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/974553f8fd06578532141df71629aa2b >[negroni] Completed 303 See Other in 125.872µs >[negroni] Started GET /blockvolumes/a596757f15215a4b45589ee382b023b2 >[negroni] Completed 200 OK in 2.276049ms >[kubeexec] DEBUG 2018/08/19 14:02:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 5min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:02:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 59min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:02:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 110.906µs >[kubeexec] DEBUG 2018/08/19 14:02:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 127.641µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 119.544µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 190.232µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 127.594µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 130.24µs >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 200 OK in 138.683µs >[kubeexec] DEBUG 2018/08/19 14:02:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_86253f54a9f617c7ecc1333fb5287181 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:681a0ad1-eff9-4f73-81fd-65e4cc1e6118", "USERNAME": "681a0ad1-eff9-4f73-81fd-65e4cc1e6118", "PASSWORD": "e779e1d7-49cc-49a4-8a43-f331e5083961", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:02:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:02:19 asynchttp.go:292: Completed job 35c885227f8cdd05d212e46d7eb172fb in 12.240773757s >[negroni] Started GET /queue/35c885227f8cdd05d212e46d7eb172fb >[negroni] Completed 303 See Other in 142.212µs >[negroni] Started GET /blockvolumes/86253f54a9f617c7ecc1333fb5287181 >[negroni] Completed 200 OK in 2.7329ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:02:36 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 92.400514ms >[asynchttp] INFO 2018/08/19 14:02:36 asynchttp.go:288: Started job 66f66eede23229147b8e925f7f4af0c0 >[heketi] INFO 2018/08/19 14:02:36 Started async operation: Create Block Volume >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 112.344µs >[cmdexec] INFO 2018/08/19 14:02:36 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:36 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:02:36 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 14:02:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:02:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 0min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:02:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:02:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:02:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:02:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:02:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:02:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 0min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:02:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:02:37 Cleaned 0 nodes from health cache >[kubeexec] DEBUG 2018/08/19 14:02:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 182.248µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:02:37 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 33.18709ms >[asynchttp] INFO 2018/08/19 14:02:37 asynchttp.go:288: Started job 96064f1e837681a2540528bda91dd4f5 >[heketi] INFO 2018/08/19 14:02:37 Started async operation: Create Block Volume >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 111.798µs >[cmdexec] INFO 2018/08/19 14:02:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:02:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 148.49µs >[kubeexec] DEBUG 2018/08/19 14:02:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 0min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:02:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 134.606µs >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 125.867µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 199.808µs >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 124.861µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 167.203µs >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 149.25µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 176.549µs >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 127.633µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 174.076µs >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 200 OK in 125.129µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 121.535µs >[kubeexec] DEBUG 2018/08/19 14:02:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fdb27da25c76b65894b197dd3845ee90 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d346fe9e-c6ff-456d-b53a-f47c54f77475", "USERNAME": "d346fe9e-c6ff-456d-b53a-f47c54f77475", "PASSWORD": "29f310a0-7293-4982-88c0-8950ead02749", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:02:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:02:44 asynchttp.go:292: Completed job 66f66eede23229147b8e925f7f4af0c0 in 7.465012576s >[kubeexec] DEBUG 2018/08/19 14:02:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:02:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:02:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/66f66eede23229147b8e925f7f4af0c0 >[negroni] Completed 303 See Other in 129.969µs >[negroni] Started GET /blockvolumes/fdb27da25c76b65894b197dd3845ee90 >[negroni] Completed 200 OK in 1.397405ms >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 122.143µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 174.89µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 140.418µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 121.031µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 128.012µs >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 200 OK in 134.564µs >[kubeexec] DEBUG 2018/08/19 14:02:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_289279dcf9b9a70543b3b43e239693cf ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:23b278ff-194a-4762-b4bf-40a5f1a9e467", "USERNAME": "23b278ff-194a-4762-b4bf-40a5f1a9e467", "PASSWORD": "067d8ae1-1adf-4c8b-a913-13e0789d46e1", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:02:50 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:02:50 asynchttp.go:292: Completed job 96064f1e837681a2540528bda91dd4f5 in 12.700796376s >[negroni] Started GET /queue/96064f1e837681a2540528bda91dd4f5 >[negroni] Completed 303 See Other in 134.49µs >[negroni] Started GET /blockvolumes/289279dcf9b9a70543b3b43e239693cf >[negroni] Completed 200 OK in 2.865514ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:03:07 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 44.930664ms >[asynchttp] INFO 2018/08/19 14:03:07 asynchttp.go:288: Started job fa82c16274e46204bbd1fc14e3abd5d5 >[heketi] INFO 2018/08/19 14:03:07 Started async operation: Create Block Volume >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 148.422µs >[cmdexec] INFO 2018/08/19 14:03:07 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:03:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:03:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 0min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:03:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 6min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 134.349µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:03:08 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 32.517233ms >[asynchttp] INFO 2018/08/19 14:03:08 asynchttp.go:288: Started job 3b8d56aa8ea21069e4db9072a36a021d >[heketi] INFO 2018/08/19 14:03:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 93.221µs >[cmdexec] INFO 2018/08/19 14:03:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 169.09µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 182.63µs >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 133.327µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 123.211µs >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 139.202µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 137.683µs >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 138.349µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 135.68µs >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 200 OK in 173.768µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 160.399µs >[kubeexec] DEBUG 2018/08/19 14:03:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_281709e57aa1afadd728a4be6aa96adc ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d844bbde-9509-496a-92eb-f027cc99afc6", "USERNAME": "d844bbde-9509-496a-92eb-f027cc99afc6", "PASSWORD": "e454dae1-2829-4b68-a449-b7d1599dde38", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:03:13 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:03:13 asynchttp.go:292: Completed job fa82c16274e46204bbd1fc14e3abd5d5 in 6.740127084s >[kubeexec] DEBUG 2018/08/19 14:03:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:03:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fa82c16274e46204bbd1fc14e3abd5d5 >[negroni] Completed 303 See Other in 168.722µs >[negroni] Started GET /blockvolumes/281709e57aa1afadd728a4be6aa96adc >[negroni] Completed 200 OK in 2.333707ms >[kubeexec] DEBUG 2018/08/19 14:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 0min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:03:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 147.887µs >[kubeexec] DEBUG 2018/08/19 14:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 6min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:03:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 174.198µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 136.378µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 171.504µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 175.087µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 119.483µs >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 200 OK in 166.852µs >[kubeexec] DEBUG 2018/08/19 14:03:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fe18ccd2995dce391cd1ef23f1e76b6a ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:ebccc978-c0a5-472f-bd79-1066039cfc17", "USERNAME": "ebccc978-c0a5-472f-bd79-1066039cfc17", "PASSWORD": "462b1065-75d4-454a-b378-c9027d97903d", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:03:20 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:03:20 asynchttp.go:292: Completed job 3b8d56aa8ea21069e4db9072a36a021d in 12.341415884s >[negroni] Started GET /queue/3b8d56aa8ea21069e4db9072a36a021d >[negroni] Completed 303 See Other in 134.844µs >[negroni] Started GET /blockvolumes/fe18ccd2995dce391cd1ef23f1e76b6a >[negroni] Completed 200 OK in 2.950565ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:03:37 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 34.449826ms >[asynchttp] INFO 2018/08/19 14:03:37 asynchttp.go:288: Started job 8557f74c0da1278af83c6189d2c890a7 >[heketi] INFO 2018/08/19 14:03:37 Started async operation: Create Block Volume >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 115.243µs >[cmdexec] INFO 2018/08/19 14:03:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 1min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:03:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:03:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 7min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:03:37 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 214.163µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:03:38 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 32.761121ms >[asynchttp] INFO 2018/08/19 14:03:38 asynchttp.go:288: Started job a0806cb5d4f8f066aa3b81ab2a4203e5 >[heketi] INFO 2018/08/19 14:03:38 Started async operation: Create Block Volume >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 117.666µs >[cmdexec] INFO 2018/08/19 14:03:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 228.401µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 125.806µs >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 190.082µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 178.205µs >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 127.127µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 171.415µs >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 126.381µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 122.727µs >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 200 OK in 126.941µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 174.01µs >[kubeexec] DEBUG 2018/08/19 14:03:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c883197305e5b27fac8353df69028d89 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:950c187a-a9be-4cf4-b433-f81232379878", "USERNAME": "950c187a-a9be-4cf4-b433-f81232379878", "PASSWORD": "09b14305-cdac-400d-8094-e4742b72f8a2", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:03:44 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:03:44 asynchttp.go:292: Completed job 8557f74c0da1278af83c6189d2c890a7 in 6.986085981s >[negroni] Started GET /queue/8557f74c0da1278af83c6189d2c890a7 >[negroni] Completed 303 See Other in 147.182µs >[negroni] Started GET /blockvolumes/c883197305e5b27fac8353df69028d89 >[negroni] Completed 200 OK in 2.315326ms >[kubeexec] DEBUG 2018/08/19 14:03:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:03:44 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 135.458µs >[kubeexec] DEBUG 2018/08/19 14:03:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 1min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:03:44 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 7min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:03:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:03:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 136.82µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 133.404µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 173.003µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 123.833µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 135.425µs >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 200 OK in 164.944µs >[kubeexec] DEBUG 2018/08/19 14:03:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_642bb5e9e73b19cbc4b61e54cb430843 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a3e69295-ec8f-4f54-b0a3-f8c78e58cbac", "USERNAME": "a3e69295-ec8f-4f54-b0a3-f8c78e58cbac", "PASSWORD": "02b083cc-2eda-4a09-be0f-c9181967f534", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:03:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:03:51 asynchttp.go:292: Completed job a0806cb5d4f8f066aa3b81ab2a4203e5 in 12.737229792s >[negroni] Started GET /queue/a0806cb5d4f8f066aa3b81ab2a4203e5 >[negroni] Completed 303 See Other in 143.156µs >[negroni] Started GET /blockvolumes/642bb5e9e73b19cbc4b61e54cb430843 >[negroni] Completed 200 OK in 2.448851ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:04:07 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 51.592704ms >[asynchttp] INFO 2018/08/19 14:04:07 asynchttp.go:288: Started job 8b1694389241b52bcabe5d6cbdc6260f >[heketi] INFO 2018/08/19 14:04:07 Started async operation: Create Block Volume >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 141.177µs >[cmdexec] INFO 2018/08/19 14:04:07 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:07 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 1min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:04:07 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:04:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 7min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:04:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 7min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 194.274µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:04:08 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 36.579701ms >[asynchttp] INFO 2018/08/19 14:04:08 asynchttp.go:288: Started job 2bb45d96dfcf3af4285eebc54a670583 >[heketi] INFO 2018/08/19 14:04:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 120.584µs >[cmdexec] INFO 2018/08/19 14:04:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 7min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:04:09 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 1min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:04:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 149.055µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 120.099µs >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 142.454µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 125.241µs >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 134.435µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 148.091µs >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 117.191µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 136.543µs >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 200 OK in 122.597µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 172.879µs >[kubeexec] DEBUG 2018/08/19 14:04:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_65fa9f2d51a01cc0c019fd6f0993082f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:b4eae0fa-eff8-4ce9-8e5a-6224364e159b", "USERNAME": "b4eae0fa-eff8-4ce9-8e5a-6224364e159b", "PASSWORD": "c536c6a7-f380-4f31-becd-c88f7fff445e", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:04:14 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:04:14 asynchttp.go:292: Completed job 8b1694389241b52bcabe5d6cbdc6260f in 6.969360945s >[negroni] Started GET /queue/8b1694389241b52bcabe5d6cbdc6260f >[negroni] Completed 303 See Other in 128.27µs >[negroni] Started GET /blockvolumes/65fa9f2d51a01cc0c019fd6f0993082f >[negroni] Completed 200 OK in 2.563072ms >[kubeexec] DEBUG 2018/08/19 14:04:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:04:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 106.861µs >[kubeexec] DEBUG 2018/08/19 14:04:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 121.637µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 177.736µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 136.139µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 126.792µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 170.855µs >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 200 OK in 130.977µs >[kubeexec] DEBUG 2018/08/19 14:04:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e306b4ad08af8a4a4b59d1eb361edc8d ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a9f24ac5-6e9e-4723-9021-95b286157347", "USERNAME": "a9f24ac5-6e9e-4723-9021-95b286157347", "PASSWORD": "1884a397-5275-4932-927d-c8219b9a4a0b", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:04:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:04:21 asynchttp.go:292: Completed job 2bb45d96dfcf3af4285eebc54a670583 in 12.732741112s >[negroni] Started GET /queue/2bb45d96dfcf3af4285eebc54a670583 >[negroni] Completed 303 See Other in 131.347µs >[negroni] Started GET /blockvolumes/e306b4ad08af8a4a4b59d1eb361edc8d >[negroni] Completed 200 OK in 2.775188ms >[heketi] INFO 2018/08/19 14:04:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:04:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:04:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:04:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:04:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:04:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 2min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:04:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:04:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:04:38 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 89.017176ms >[asynchttp] INFO 2018/08/19 14:04:38 asynchttp.go:288: Started job e6835b3aeab8fb9391eb2f46f72b10f0 >[heketi] INFO 2018/08/19 14:04:38 Started async operation: Create Block Volume >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 126.109µs >[cmdexec] INFO 2018/08/19 14:04:38 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:04:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:04:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 2min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:04:38 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 119.55µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:04:39 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 42.192725ms >[asynchttp] INFO 2018/08/19 14:04:39 asynchttp.go:288: Started job 8d3829dc300928d4ec1f81dbcca6766a >[heketi] INFO 2018/08/19 14:04:39 Started async operation: Create Block Volume >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 114.145µs >[cmdexec] INFO 2018/08/19 14:04:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 149.449µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 172.612µs >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 157.185µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 206.14µs >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 146.006µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 187.813µs >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 159.785µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 116.911µs >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 200 OK in 128.262µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 138.404µs >[kubeexec] DEBUG 2018/08/19 14:04:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ff37ad51414fff00fe9b94d1102363cc ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:baee8a97-cb02-4670-8850-74ddfd788a14", "USERNAME": "baee8a97-cb02-4670-8850-74ddfd788a14", "PASSWORD": "855e7d0a-162e-403f-ad67-3745304834cf", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:04:45 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:04:45 asynchttp.go:292: Completed job e6835b3aeab8fb9391eb2f46f72b10f0 in 6.968691336s >[negroni] Started GET /queue/e6835b3aeab8fb9391eb2f46f72b10f0 >[negroni] Completed 303 See Other in 161.615µs >[negroni] Started GET /blockvolumes/ff37ad51414fff00fe9b94d1102363cc >[negroni] Completed 200 OK in 2.759866ms >[kubeexec] DEBUG 2018/08/19 14:04:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:04:45 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 131.363µs >[kubeexec] DEBUG 2018/08/19 14:04:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 2min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:04:45 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:04:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:04:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 184.319µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 134.96µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 202.75µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 133.257µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 174.665µs >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 200 OK in 186.413µs >[kubeexec] DEBUG 2018/08/19 14:04:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_916c49610dba09c5a3260bc0fbf09408 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:cd7465d0-3fa8-48b1-9557-2a8d7a412a67", "USERNAME": "cd7465d0-3fa8-48b1-9557-2a8d7a412a67", "PASSWORD": "6a3b333e-55e1-484e-9cd7-eecdd2ba1d83", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:04:52 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:04:52 asynchttp.go:292: Completed job 8d3829dc300928d4ec1f81dbcca6766a in 12.799284783s >[negroni] Started GET /queue/8d3829dc300928d4ec1f81dbcca6766a >[negroni] Completed 303 See Other in 137.955µs >[negroni] Started GET /blockvolumes/916c49610dba09c5a3260bc0fbf09408 >[negroni] Completed 200 OK in 2.264639ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:05:08 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 37.666115ms >[asynchttp] INFO 2018/08/19 14:05:08 asynchttp.go:288: Started job 4d49a288ce8adfb87c018aa6d44017f4 >[heketi] INFO 2018/08/19 14:05:08 Started async operation: Create Block Volume >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 98.782µs >[cmdexec] INFO 2018/08/19 14:05:08 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 2min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:05:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:05:08 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:05:08 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:08 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 8min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 170.947µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:05:09 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 44.068092ms >[asynchttp] INFO 2018/08/19 14:05:09 asynchttp.go:288: Started job dd79de36de288621f8b234ac76232244 >[heketi] INFO 2018/08/19 14:05:09 Started async operation: Create Block Volume >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 147.898µs >[cmdexec] INFO 2018/08/19 14:05:09 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 8min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:05:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 144.089µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 131.325µs >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 136.979µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 126.169µs >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 141.063µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 142.652µs >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 129.628µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 141.653µs >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 132.003µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 162.591µs >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 200 OK in 136.077µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 127.551µs >[kubeexec] DEBUG 2018/08/19 14:05:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_4f4f6d013fcbabea58c955e99f55b39a ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d8bf553b-00f5-4492-b9b4-0f007e97109b", "USERNAME": "d8bf553b-00f5-4492-b9b4-0f007e97109b", "PASSWORD": "9e84cb4b-435a-4599-95d1-ca2cb3aa996c", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:05:15 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:05:15 asynchttp.go:292: Completed job 4d49a288ce8adfb87c018aa6d44017f4 in 7.345645416s >[kubeexec] DEBUG 2018/08/19 14:05:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:05:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 2min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:05:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4d49a288ce8adfb87c018aa6d44017f4 >[negroni] Completed 303 See Other in 134.137µs >[negroni] Started GET /blockvolumes/4f4f6d013fcbabea58c955e99f55b39a >[negroni] Completed 200 OK in 1.4489ms >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 123.714µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 183.485µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 118.06µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 121.835µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 124.771µs >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 200 OK in 124.482µs >[kubeexec] DEBUG 2018/08/19 14:05:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0a7bc7ade6ef12b836b4a0d92cdc866e ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:d9351108-f302-4630-8025-941af9c58abc", "USERNAME": "d9351108-f302-4630-8025-941af9c58abc", "PASSWORD": "14761948-f151-42a5-9bfe-61d917c272a7", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:05:22 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:05:22 asynchttp.go:292: Completed job dd79de36de288621f8b234ac76232244 in 13.00424433s >[negroni] Started GET /queue/dd79de36de288621f8b234ac76232244 >[negroni] Completed 303 See Other in 157.717µs >[negroni] Started GET /blockvolumes/0a7bc7ade6ef12b836b4a0d92cdc866e >[negroni] Completed 200 OK in 2.040847ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:05:38 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 38.465502ms >[asynchttp] INFO 2018/08/19 14:05:38 asynchttp.go:288: Started job b0ba0d5fce46b229f338b85e4b2f3f69 >[heketi] INFO 2018/08/19 14:05:38 Started async operation: Create Block Volume >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 107.375µs >[cmdexec] INFO 2018/08/19 14:05:38 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:38 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 9min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:05:38 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 3min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:05:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:05:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 138.633µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:05:39 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 39.546756ms >[asynchttp] INFO 2018/08/19 14:05:39 asynchttp.go:288: Started job 1d5876c8ec3c134e9793842d192c43e1 >[heketi] INFO 2018/08/19 14:05:39 Started async operation: Create Block Volume >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 122.263µs >[cmdexec] INFO 2018/08/19 14:05:39 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 9min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:05:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 173.951µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 170.354µs >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 123.042µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 124.938µs >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 128.493µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 191.195µs >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 129.08µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 127.436µs >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 191.199µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 198.874µs >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 200 OK in 155.527µs >[kubeexec] DEBUG 2018/08/19 14:05:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_7bb04582bbf8ff583076b5dd7251ae2c ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:3a80e30e-d68d-4955-81f2-5b6c4f166521", "USERNAME": "3a80e30e-d68d-4955-81f2-5b6c4f166521", "PASSWORD": "ae4dfbda-ad37-47f4-8a88-da5e18bd1f4f", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:05:45 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:05:45 asynchttp.go:292: Completed job b0ba0d5fce46b229f338b85e4b2f3f69 in 7.159249713s >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 119.613µs >[kubeexec] DEBUG 2018/08/19 14:05:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:05:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 3min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:05:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:05:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b0ba0d5fce46b229f338b85e4b2f3f69 >[negroni] Completed 303 See Other in 187.794µs >[negroni] Started GET /blockvolumes/7bb04582bbf8ff583076b5dd7251ae2c >[negroni] Completed 200 OK in 1.538803ms >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 166.635µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 172.162µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 149.884µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 187.422µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 130.298µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 119.67µs >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 200 OK in 134.714µs >[kubeexec] DEBUG 2018/08/19 14:05:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_bc464dce766e93d493cb407fff7dea26 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:5c4c1614-6cb9-4ff8-b077-9fc980e6e7bb", "USERNAME": "5c4c1614-6cb9-4ff8-b077-9fc980e6e7bb", "PASSWORD": "4f0e1986-7b39-4482-90a8-f9790ece07d7", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:05:53 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:05:53 asynchttp.go:292: Completed job 1d5876c8ec3c134e9793842d192c43e1 in 13.095536509s >[negroni] Started GET /queue/1d5876c8ec3c134e9793842d192c43e1 >[negroni] Completed 303 See Other in 147.786µs >[negroni] Started GET /blockvolumes/bc464dce766e93d493cb407fff7dea26 >[negroni] Completed 200 OK in 2.63927ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:06:09 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 41.039868ms >[asynchttp] INFO 2018/08/19 14:06:09 asynchttp.go:288: Started job bd671caddb71e06e656cd7056b9b6878 >[heketi] INFO 2018/08/19 14:06:09 Started async operation: Create Block Volume >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 97.302µs >[cmdexec] INFO 2018/08/19 14:06:09 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 9min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:06:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:06:09 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 3min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:06:09 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 9min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 125.314µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:06:10 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 37.302654ms >[asynchttp] INFO 2018/08/19 14:06:10 asynchttp.go:288: Started job d48fc593d5b54a924c612041b23d8ccd >[heketi] INFO 2018/08/19 14:06:10 Started async operation: Create Block Volume >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 111.552µs >[cmdexec] INFO 2018/08/19 14:06:10 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 9min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:06:10 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 3min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:06:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 133.424µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 188.304µs >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 141.839µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 169.547µs >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 141.821µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 178.09µs >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 128.245µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 126.376µs >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 142.84µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 188.879µs >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 200 OK in 122.187µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 136.513µs >[kubeexec] DEBUG 2018/08/19 14:06:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e780d718e09fbe2f60c7e0a1e8e6052c ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f143e509-148b-4529-a281-a3d8211ad2cb", "USERNAME": "f143e509-148b-4529-a281-a3d8211ad2cb", "PASSWORD": "61821372-3872-4987-9880-04f3505a9485", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[kubeexec] DEBUG 2018/08/19 14:06:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:06:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[heketi] INFO 2018/08/19 14:06:16 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:06:16 asynchttp.go:292: Completed job bd671caddb71e06e656cd7056b9b6878 in 7.727902028s >[kubeexec] DEBUG 2018/08/19 14:06:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/bd671caddb71e06e656cd7056b9b6878 >[negroni] Completed 303 See Other in 136.255µs >[negroni] Started GET /blockvolumes/e780d718e09fbe2f60c7e0a1e8e6052c >[negroni] Completed 200 OK in 2.852082ms >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 158.774µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 185.359µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 161.39µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 123.569µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 114.711µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 130.55µs >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 200 OK in 133.264µs >[kubeexec] DEBUG 2018/08/19 14:06:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_bd2cde55351555adaf9d6b7700970b8d ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:47df8065-a208-4ecb-90fc-0a35bd60201f", "USERNAME": "47df8065-a208-4ecb-90fc-0a35bd60201f", "PASSWORD": "649b3118-806d-4ee1-bdc1-08bc418c9416", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:06:23 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:06:23 asynchttp.go:292: Completed job d48fc593d5b54a924c612041b23d8ccd in 13.472778178s >[negroni] Started GET /queue/d48fc593d5b54a924c612041b23d8ccd >[negroni] Completed 303 See Other in 180.011µs >[negroni] Started GET /blockvolumes/bd2cde55351555adaf9d6b7700970b8d >[negroni] Completed 200 OK in 5.468967ms >[heketi] INFO 2018/08/19 14:06:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:06:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:06:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:06:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:06:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:06:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 4min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:06:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:06:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:06:39 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 91.418677ms >[asynchttp] INFO 2018/08/19 14:06:39 asynchttp.go:288: Started job 06b74cf981b5ef31885a02dd026d791f >[heketi] INFO 2018/08/19 14:06:39 Started async operation: Create Block Volume >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 127.952µs >[cmdexec] INFO 2018/08/19 14:06:39 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:06:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:06:39 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:39 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 4min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:06:39 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 131.86µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:06:40 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 35.825417ms >[asynchttp] INFO 2018/08/19 14:06:40 asynchttp.go:288: Started job 6a58d0b00b0bc572880ea449d2ca0bb1 >[heketi] INFO 2018/08/19 14:06:40 Started async operation: Create Block Volume >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 91.806µs >[cmdexec] INFO 2018/08/19 14:06:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:06:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 192.392µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 114.313µs >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 141.711µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 140.146µs >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 152.957µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 194.105µs >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 138.147µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 115.637µs >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 206.625µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 185.383µs >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 200 OK in 192.169µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 127.85µs >[kubeexec] DEBUG 2018/08/19 14:06:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_bbc102145e025860159e245a1b18c302 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:8fca30bb-453b-413b-8bd2-9bae5eb1e241", "USERNAME": "8fca30bb-453b-413b-8bd2-9bae5eb1e241", "PASSWORD": "c5733fde-c1f0-4caf-b1e5-cc562a8ad478", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:06:46 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:06:46 asynchttp.go:292: Completed job 06b74cf981b5ef31885a02dd026d791f in 7.459906626s >[kubeexec] DEBUG 2018/08/19 14:06:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:06:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 4min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:06:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:06:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/06b74cf981b5ef31885a02dd026d791f >[negroni] Completed 303 See Other in 127.827µs >[negroni] Started GET /blockvolumes/bbc102145e025860159e245a1b18c302 >[negroni] Completed 200 OK in 1.475918ms >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 137.683µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 164.742µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 136.001µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 121.597µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 185.453µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 130.649µs >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 200 OK in 168.227µs >[kubeexec] DEBUG 2018/08/19 14:06:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ce6badf41b68bdea2033ff765fc693f9 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:0b1db0ef-aa7a-47c6-b39c-1d6a6d4fda46", "USERNAME": "0b1db0ef-aa7a-47c6-b39c-1d6a6d4fda46", "PASSWORD": "e43a4c25-3c31-4243-823e-99928de154de", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:06:54 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:06:54 asynchttp.go:292: Completed job 6a58d0b00b0bc572880ea449d2ca0bb1 in 13.658568014s >[negroni] Started GET /queue/6a58d0b00b0bc572880ea449d2ca0bb1 >[negroni] Completed 303 See Other in 216.634µs >[negroni] Started GET /blockvolumes/ce6badf41b68bdea2033ff765fc693f9 >[negroni] Completed 200 OK in 2.562214ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:07:09 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 44.509783ms >[asynchttp] INFO 2018/08/19 14:07:09 asynchttp.go:288: Started job 741a33fd0741ea24c76deb171716de76 >[heketi] INFO 2018/08/19 14:07:09 Started async operation: Create Block Volume >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 116.986µs >[cmdexec] INFO 2018/08/19 14:07:09 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:09 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 4min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:07:09 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:07:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:07:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 10min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 119.412µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:07:10 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 35.429553ms >[asynchttp] INFO 2018/08/19 14:07:10 asynchttp.go:288: Started job 93e31283c707c89a3ed289b43cc958c0 >[heketi] INFO 2018/08/19 14:07:10 Started async operation: Create Block Volume >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 171.914µs >[cmdexec] INFO 2018/08/19 14:07:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 167.342µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 127.407µs >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 121.161µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 140.752µs >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 175.849µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 242.797µs >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 207.739µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 129.984µs >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 150.22µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 176.657µs >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 200 OK in 185.486µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 141.886µs >[kubeexec] DEBUG 2018/08/19 14:07:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_9a4e25906807000fc09ec4520e31ca8b ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:f334c356-5f69-492d-af4e-36cf6cf0e898", "USERNAME": "f334c356-5f69-492d-af4e-36cf6cf0e898", "PASSWORD": "0265afcd-ca75-4471-b9f5-7efb6de90fbb", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:07:17 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:07:17 asynchttp.go:292: Completed job 741a33fd0741ea24c76deb171716de76 in 7.525033735s >[kubeexec] DEBUG 2018/08/19 14:07:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:07:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 4min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:07:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 10min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:07:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/741a33fd0741ea24c76deb171716de76 >[negroni] Completed 303 See Other in 125.348µs >[negroni] Started GET /blockvolumes/9a4e25906807000fc09ec4520e31ca8b >[negroni] Completed 200 OK in 1.454346ms >[kubeexec] DEBUG 2018/08/19 14:07:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 148.955µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 124.339µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 271.128µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 149.202µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 123.071µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 224.089µs >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 200 OK in 128.191µs >[kubeexec] DEBUG 2018/08/19 14:07:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_189fbdd5c671c0c8437b7bfc1791d321 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:91ea29f2-4189-46d3-a536-2e522f87649d", "USERNAME": "91ea29f2-4189-46d3-a536-2e522f87649d", "PASSWORD": "0a4a89ee-eb8b-42a1-b73b-10d36f89c66e", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:07:24 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:07:24 asynchttp.go:292: Completed job 93e31283c707c89a3ed289b43cc958c0 in 13.672187992s >[negroni] Started GET /queue/93e31283c707c89a3ed289b43cc958c0 >[negroni] Completed 303 See Other in 187.999µs >[negroni] Started GET /blockvolumes/189fbdd5c671c0c8437b7bfc1791d321 >[negroni] Completed 200 OK in 2.369478ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:07:40 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 36.305118ms >[asynchttp] INFO 2018/08/19 14:07:40 asynchttp.go:288: Started job 7a8bb63d6b0c3363d2409ef45b59e2ba >[heketi] INFO 2018/08/19 14:07:40 Started async operation: Create Block Volume >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 115.079µs >[cmdexec] INFO 2018/08/19 14:07:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:07:40 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 11min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:07:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 5min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:07:40 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:40 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 131.707µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:07:41 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 35.739629ms >[asynchttp] INFO 2018/08/19 14:07:41 asynchttp.go:288: Started job 8ac2fc9ec84225c1405ca72dc432efaa >[heketi] INFO 2018/08/19 14:07:41 Started async operation: Create Block Volume >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 152.979µs >[cmdexec] INFO 2018/08/19 14:07:41 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 5min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:07:41 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 11min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:07:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 178.656µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 201.036µs >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 164.839µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 165.565µs >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 165.073µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 154.259µs >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 121.799µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 117.45µs >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 126.411µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 121.154µs >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 200 OK in 122.313µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 172.127µs >[kubeexec] DEBUG 2018/08/19 14:07:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_298d8cd2483ddf339d1e56aec373d8a1 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:af08290e-6b63-42a5-b3db-ef924c0c0a90", "USERNAME": "af08290e-6b63-42a5-b3db-ef924c0c0a90", "PASSWORD": "e059e8b2-72dc-49b0-8ac3-c0c32c5fba52", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:07:47 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:07:47 asynchttp.go:292: Completed job 7a8bb63d6b0c3363d2409ef45b59e2ba in 7.736050941s >[kubeexec] DEBUG 2018/08/19 14:07:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:07:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:07:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 11min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7a8bb63d6b0c3363d2409ef45b59e2ba >[negroni] Completed 303 See Other in 166.639µs >[negroni] Started GET /blockvolumes/298d8cd2483ddf339d1e56aec373d8a1 >[negroni] Completed 200 OK in 1.519198ms >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 144.024µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 159.961µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 130.475µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 131.7µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 170.194µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 177.695µs >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 200 OK in 161.328µs >[kubeexec] DEBUG 2018/08/19 14:07:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d5c23911ae6623a069250f15aa372949 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:947057f9-b737-4c51-85d7-8754b3f2e386", "USERNAME": "947057f9-b737-4c51-85d7-8754b3f2e386", "PASSWORD": "1da50447-16bb-4d4e-815f-e44322fe7b20", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:07:55 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:07:55 asynchttp.go:292: Completed job 8ac2fc9ec84225c1405ca72dc432efaa in 13.903496056s >[negroni] Started GET /queue/8ac2fc9ec84225c1405ca72dc432efaa >[negroni] Completed 303 See Other in 129.861µs >[negroni] Started GET /blockvolumes/d5c23911ae6623a069250f15aa372949 >[negroni] Completed 200 OK in 2.674038ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:08:10 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 41.697602ms >[asynchttp] INFO 2018/08/19 14:08:10 asynchttp.go:288: Started job 0045f138a0d876a5e18c7d33e815ef31 >[heketi] INFO 2018/08/19 14:08:10 Started async operation: Create Block Volume >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 141.029µs >[cmdexec] INFO 2018/08/19 14:08:10 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 5min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:08:10 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:10 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:08:10 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 11min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:08:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 135.226µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:08:11 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 40.766557ms >[asynchttp] INFO 2018/08/19 14:08:11 asynchttp.go:288: Started job ee698d7b3e640e13e4f14e2bc749e8f0 >[heketi] INFO 2018/08/19 14:08:11 Started async operation: Create Block Volume >[cmdexec] INFO 2018/08/19 14:08:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 114.906µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 161.359µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 287.865µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 193.644µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 141.443µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 168.431µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 159.587µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 137.998µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 123.777µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 172.11µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 178.887µs >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 200 OK in 123.025µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 182.546µs >[kubeexec] DEBUG 2018/08/19 14:08:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_3d345b69dee6ab12584e99d520bb773e ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:06baffd9-94c9-43e2-a119-1736f0ea732f", "USERNAME": "06baffd9-94c9-43e2-a119-1736f0ea732f", "PASSWORD": "7ad049f6-60c3-4354-9d55-280212cf9ee7", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:08:18 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:08:18 asynchttp.go:292: Completed job 0045f138a0d876a5e18c7d33e815ef31 in 7.46081642s >[kubeexec] DEBUG 2018/08/19 14:08:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:08:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 5min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:08:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 11min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:08:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0045f138a0d876a5e18c7d33e815ef31 >[negroni] Completed 303 See Other in 129.986µs >[negroni] Started GET /blockvolumes/3d345b69dee6ab12584e99d520bb773e >[negroni] Completed 200 OK in 1.52138ms >[kubeexec] DEBUG 2018/08/19 14:08:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 164.035µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 141.631µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 127.718µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 132.915µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 188.061µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 175.356µs >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 200 OK in 198.49µs >[kubeexec] DEBUG 2018/08/19 14:08:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_b9eb0dd0fdf3ce3cdd755dd8f0ae3e45 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:47278dae-7149-4113-8539-54a4769f9273", "USERNAME": "47278dae-7149-4113-8539-54a4769f9273", "PASSWORD": "6992d50a-7dc0-4deb-a284-1915db71b003", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:08:25 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:08:25 asynchttp.go:292: Completed job ee698d7b3e640e13e4f14e2bc749e8f0 in 14.005768052s >[negroni] Started GET /queue/ee698d7b3e640e13e4f14e2bc749e8f0 >[negroni] Completed 303 See Other in 142.368µs >[negroni] Started GET /blockvolumes/b9eb0dd0fdf3ce3cdd755dd8f0ae3e45 >[negroni] Completed 200 OK in 2.760174ms >[heketi] INFO 2018/08/19 14:08:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:08:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:08:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:08:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:08:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:08:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:08:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:08:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:08:40 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 39.36059ms >[asynchttp] INFO 2018/08/19 14:08:40 asynchttp.go:288: Started job d2df9cac2787afbfee9e705b744217d5 >[heketi] INFO 2018/08/19 14:08:40 Started async operation: Create Block Volume >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 180.023µs >[cmdexec] INFO 2018/08/19 14:08:40 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:08:41 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:08:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:08:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 174.698µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:08:42 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 43.767278ms >[asynchttp] INFO 2018/08/19 14:08:42 asynchttp.go:288: Started job 99748f7ec0a94acac92ccbb52ba285fc >[heketi] INFO 2018/08/19 14:08:42 Started async operation: Create Block Volume >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 132.4µs >[cmdexec] INFO 2018/08/19 14:08:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 130.738µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 166.365µs >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 189.897µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 126.779µs >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 142.931µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 147.523µs >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 216.637µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 180.511µs >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 136.435µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 155.077µs >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 200 OK in 151.839µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 186.296µs >[kubeexec] DEBUG 2018/08/19 14:08:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_92b966e37f76ca0d2092df19542c8b72 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e51d9550-1633-48ae-a20f-8b7022c26f56", "USERNAME": "e51d9550-1633-48ae-a20f-8b7022c26f56", "PASSWORD": "3e8e5d1a-d3fe-472d-b44e-f2c457fe3d77", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:08:48 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:08:48 asynchttp.go:292: Completed job d2df9cac2787afbfee9e705b744217d5 in 7.689302092s >[kubeexec] DEBUG 2018/08/19 14:08:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:08:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:08:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:08:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/d2df9cac2787afbfee9e705b744217d5 >[negroni] Completed 303 See Other in 183.566µs >[negroni] Started GET /blockvolumes/92b966e37f76ca0d2092df19542c8b72 >[negroni] Completed 200 OK in 2.086682ms >[kubeexec] DEBUG 2018/08/19 14:08:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:08:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 183.849µs >[kubeexec] DEBUG 2018/08/19 14:08:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 12min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 337.605µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 166.682µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 184.385µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 157.629µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 134.16µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 132.347µs >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 200 OK in 150.248µs >[kubeexec] DEBUG 2018/08/19 14:08:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0c2b1a47df4424733021aa90d1b3a16b ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:dca641eb-d67c-4fee-8307-0ddd97931d3d", "USERNAME": "dca641eb-d67c-4fee-8307-0ddd97931d3d", "PASSWORD": "8fe4f2c2-7e11-40d9-8c15-cdad08ac8c6a", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:08:56 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:08:56 asynchttp.go:292: Completed job 99748f7ec0a94acac92ccbb52ba285fc in 14.299878184s >[negroni] Started GET /queue/99748f7ec0a94acac92ccbb52ba285fc >[negroni] Completed 303 See Other in 145.898µs >[negroni] Started GET /blockvolumes/0c2b1a47df4424733021aa90d1b3a16b >[negroni] Completed 200 OK in 2.215052ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:09:11 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 38.985771ms >[asynchttp] INFO 2018/08/19 14:09:11 asynchttp.go:288: Started job 259d5174e79e360b4b84af4476dd5a6d >[heketi] INFO 2018/08/19 14:09:11 Started async operation: Create Block Volume >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 114.392µs >[cmdexec] INFO 2018/08/19 14:09:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:09:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 12min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:09:11 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:09:11 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:11 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 202.358µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:09:12 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 80.361374ms >[asynchttp] INFO 2018/08/19 14:09:12 asynchttp.go:288: Started job c203350113f577dbce62d28986f0d84b >[heketi] INFO 2018/08/19 14:09:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 102.217µs >[cmdexec] INFO 2018/08/19 14:09:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 118.822µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 123.426µs >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 134.69µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 159.85µs >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 116.999µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 159.588µs >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 158.849µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 183.016µs >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 120.208µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 123.089µs >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 200 OK in 129.1µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 123.025µs >[kubeexec] DEBUG 2018/08/19 14:09:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_8dc50a0ab3e2204e1af2f630d00e3576 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:06ddb823-2047-4cab-8974-c96ad1f3d017", "USERNAME": "06ddb823-2047-4cab-8974-c96ad1f3d017", "PASSWORD": "15348a09-f878-4156-a7e1-76cf1c1ff86b", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.46.152:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:09:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:09:19 asynchttp.go:292: Completed job 259d5174e79e360b4b84af4476dd5a6d in 7.738074504s >[kubeexec] DEBUG 2018/08/19 14:09:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:09:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/259d5174e79e360b4b84af4476dd5a6d >[negroni] Completed 303 See Other in 198.547µs >[negroni] Started GET /blockvolumes/8dc50a0ab3e2204e1af2f630d00e3576 >[negroni] Completed 200 OK in 3.437614ms >[kubeexec] DEBUG 2018/08/19 14:09:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 6min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:09:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:09:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 257.29µs >[kubeexec] DEBUG 2018/08/19 14:09:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 164.086µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 194.105µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 166.162µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 136.001µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 161.148µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 209.025µs >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 200 OK in 158.629µs >[kubeexec] DEBUG 2018/08/19 14:09:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_8b07b802700756032e4e37284aef962d ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fc843844-29e2-4a12-9337-e4b59373ad40", "USERNAME": "fc843844-29e2-4a12-9337-e4b59373ad40", "PASSWORD": "a6edf73c-a54d-4fd1-b9b4-d536c45473c9", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:09:26 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:09:26 asynchttp.go:292: Completed job c203350113f577dbce62d28986f0d84b in 14.220876459s >[negroni] Started GET /queue/c203350113f577dbce62d28986f0d84b >[negroni] Completed 303 See Other in 164.826µs >[negroni] Started GET /blockvolumes/8b07b802700756032e4e37284aef962d >[negroni] Completed 200 OK in 2.643361ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:09:41 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 41.813588ms >[asynchttp] INFO 2018/08/19 14:09:41 asynchttp.go:288: Started job 4f77b26f268307e200bc50473c44e834 >[heketi] INFO 2018/08/19 14:09:41 Started async operation: Create Block Volume >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 134.673µs >[cmdexec] INFO 2018/08/19 14:09:41 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:09:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:09:41 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:41 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:09:41 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 212.512µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:09:42 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 40.785237ms >[asynchttp] INFO 2018/08/19 14:09:42 asynchttp.go:288: Started job ea432b192a5090b1f379b4175497e8c6 >[heketi] INFO 2018/08/19 14:09:42 Started async operation: Create Block Volume >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 112.237µs >[cmdexec] INFO 2018/08/19 14:09:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:09:42 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:09:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:09:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 175.46µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 122.46µs >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 171.612µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 253.405µs >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 299.152µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 134.026µs >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 184.283µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 116.314µs >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 142.613µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 127.273µs >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 200 OK in 208.774µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 154.983µs >[kubeexec] DEBUG 2018/08/19 14:09:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c677ffa6a44d1a1dd2a481300fdace43 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:1d62c447-b31a-46e3-ad6e-23eb55613451", "USERNAME": "1d62c447-b31a-46e3-ad6e-23eb55613451", "PASSWORD": "3182ac8e-3a5a-4ba2-a96e-875012bab0ec", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:09:49 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:09:49 asynchttp.go:292: Completed job 4f77b26f268307e200bc50473c44e834 in 7.79128148s >[kubeexec] DEBUG 2018/08/19 14:09:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:09:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4f77b26f268307e200bc50473c44e834 >[negroni] Completed 303 See Other in 136.455µs >[negroni] Started GET /blockvolumes/c677ffa6a44d1a1dd2a481300fdace43 >[negroni] Completed 200 OK in 1.460982ms >[kubeexec] DEBUG 2018/08/19 14:09:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 13min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 135.81µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 147.676µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 153.416µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 152.464µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 128.913µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 144.837µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 124.039µs >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 200 OK in 143.284µs >[kubeexec] DEBUG 2018/08/19 14:09:56 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_af369152e8ec91052e6322e6feb6f5b0 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:6090c6b6-06a9-45b9-af3c-8c3419bf074a", "USERNAME": "6090c6b6-06a9-45b9-af3c-8c3419bf074a", "PASSWORD": "6607fa94-df69-4518-9e25-49a640d9865b", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:09:56 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:09:56 asynchttp.go:292: Completed job ea432b192a5090b1f379b4175497e8c6 in 14.173325992s >[negroni] Started GET /queue/ea432b192a5090b1f379b4175497e8c6 >[negroni] Completed 303 See Other in 144.024µs >[negroni] Started GET /blockvolumes/af369152e8ec91052e6322e6feb6f5b0 >[negroni] Completed 200 OK in 2.24358ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:10:11 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 37.788998ms >[asynchttp] INFO 2018/08/19 14:10:11 asynchttp.go:288: Started job de4fb6df109c18f35295bac7e2f2de54 >[heketi] INFO 2018/08/19 14:10:11 Started async operation: Create Block Volume >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 99.221µs >[cmdexec] INFO 2018/08/19 14:10:11 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:10:12 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:10:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:10:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 123.587µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:10:13 Free size 0 is less than the requested block volume size 1 >[asynchttp] INFO 2018/08/19 14:10:13 asynchttp.go:288: Started job 59a27cdeca10113a1bb79e1423a5fccb >[heketi] INFO 2018/08/19 14:10:13 Started async operation: Create Block Volume >[negroni] Completed 202 Accepted in 78.588294ms >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 129.347µs >[cmdexec] INFO 2018/08/19 14:10:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 7min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:10:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 13min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:10:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 123.284µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 135.423µs >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 122.798µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 232.866µs >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 126.846µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 126.75µs >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 159.509µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 155.099µs >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 197.155µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 139.188µs >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 200 OK in 168.776µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 145.189µs >[kubeexec] DEBUG 2018/08/19 14:10:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_ef611a9dc7499003aaad5df8016fcb4f ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:5cbce575-8eac-494b-8988-81a3e57649c7", "USERNAME": "5cbce575-8eac-494b-8988-81a3e57649c7", "PASSWORD": "7a5e9d61-1aa6-43ee-9274-05a418901aba", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.54:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:10:19 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:10:19 asynchttp.go:292: Completed job de4fb6df109c18f35295bac7e2f2de54 in 7.72016835s >[kubeexec] DEBUG 2018/08/19 14:10:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:10:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/de4fb6df109c18f35295bac7e2f2de54 >[negroni] Completed 303 See Other in 105.505µs >[negroni] Started GET /blockvolumes/ef611a9dc7499003aaad5df8016fcb4f >[negroni] Completed 200 OK in 1.788774ms >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 124.076µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 168.358µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 127.071µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 150.538µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 171.604µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 123.855µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 150.568µs >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 200 OK in 134.862µs >[kubeexec] DEBUG 2018/08/19 14:10:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_1edd80a52bf182cb38764683d8663e9a ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:98531f9a-79c0-4dbd-a2b7-bd6466f60810", "USERNAME": "98531f9a-79c0-4dbd-a2b7-bd6466f60810", "PASSWORD": "a5f0a55b-25e1-4ecc-abdb-199ed2fd63c6", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:10:27 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:10:27 asynchttp.go:292: Completed job 59a27cdeca10113a1bb79e1423a5fccb in 14.149473848s >[negroni] Started GET /queue/59a27cdeca10113a1bb79e1423a5fccb >[negroni] Completed 303 See Other in 150.675µs >[negroni] Started GET /blockvolumes/1edd80a52bf182cb38764683d8663e9a >[negroni] Completed 200 OK in 2.26351ms >[heketi] INFO 2018/08/19 14:10:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:10:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:10:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:10:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:10:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:10:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:10:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:10:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:10:42 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 38.538365ms >[asynchttp] INFO 2018/08/19 14:10:42 asynchttp.go:288: Started job 0729ec836264c97f10493eeafd7a8101 >[heketi] INFO 2018/08/19 14:10:42 Started async operation: Create Block Volume >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 150.341µs >[cmdexec] INFO 2018/08/19 14:10:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:10:42 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:10:42 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:10:42 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:42 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 159.666µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:10:43 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 50.415577ms >[asynchttp] INFO 2018/08/19 14:10:43 asynchttp.go:288: Started job c6bde67470df187734f323a2c61dbabb >[heketi] INFO 2018/08/19 14:10:43 Started async operation: Create Block Volume >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 103.916µs >[cmdexec] INFO 2018/08/19 14:10:43 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:10:43 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:10:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:10:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 153.142µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 148.033µs >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 118.224µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 204.621µs >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 128.071µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 155.13µs >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 156.396µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 155.595µs >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 161.207µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 216.419µs >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 200 OK in 120.67µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 175.318µs >[kubeexec] DEBUG 2018/08/19 14:10:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_16f613f22ef79491034df20837ec25eb ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:af0b6e88-c1ef-4aab-9784-9a79729ce925", "USERNAME": "af0b6e88-c1ef-4aab-9784-9a79729ce925", "PASSWORD": "7fd95888-df9c-409f-af15-b602b27ce874", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:10:50 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:10:50 asynchttp.go:292: Completed job 0729ec836264c97f10493eeafd7a8101 in 7.77698327s >[kubeexec] DEBUG 2018/08/19 14:10:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:10:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0729ec836264c97f10493eeafd7a8101 >[negroni] Completed 303 See Other in 139.312µs >[negroni] Started GET /blockvolumes/16f613f22ef79491034df20837ec25eb >[negroni] Completed 200 OK in 1.684876ms >[kubeexec] DEBUG 2018/08/19 14:10:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 14min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 128.083µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 219.774µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 175.424µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 126.092µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 170.213µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 149.754µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 118.693µs >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 200 OK in 125.827µs >[kubeexec] DEBUG 2018/08/19 14:10:57 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_2b020a18e78a5c6b0508f86e2d566131 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:1cebd551-d1e3-48a1-90d0-3ec33cc93816", "USERNAME": "1cebd551-d1e3-48a1-90d0-3ec33cc93816", "PASSWORD": "c78f40a7-f0b6-4161-99c4-c73df7bcc48a", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:10:57 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:10:57 asynchttp.go:292: Completed job c6bde67470df187734f323a2c61dbabb in 14.210903716s >[negroni] Started GET /queue/c6bde67470df187734f323a2c61dbabb >[negroni] Completed 303 See Other in 126.725µs >[negroni] Started GET /blockvolumes/2b020a18e78a5c6b0508f86e2d566131 >[negroni] Completed 200 OK in 2.69439ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:11:12 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 139.297879ms >[asynchttp] INFO 2018/08/19 14:11:12 asynchttp.go:288: Started job 4ca8dfec34ba07abb49a78c4557403ef >[heketi] INFO 2018/08/19 14:11:12 Started async operation: Create Block Volume >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 156.195µs >[cmdexec] INFO 2018/08/19 14:11:12 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:12 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:11:12 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:11:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:11:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 132.061µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:11:13 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 56.35721ms >[asynchttp] INFO 2018/08/19 14:11:13 asynchttp.go:288: Started job 4f8b27df8dbde661d37a3bbcdb6de313 >[heketi] INFO 2018/08/19 14:11:13 Started async operation: Create Block Volume >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 142.828µs >[cmdexec] INFO 2018/08/19 14:11:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 8min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:11:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 14min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:11:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 128.42µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 129.674µs >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 128.208µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 141.871µs >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 125.53µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 180.698µs >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 126.611µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 120.867µs >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 148.32µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 131.654µs >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 133.144µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 120.095µs >[kubeexec] DEBUG 2018/08/19 14:11:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d2d79dee195dcaaa113173558896208d ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:c289b6e9-8019-4b96-9889-63ba29e36338", "USERNAME": "c289b6e9-8019-4b96-9889-63ba29e36338", "PASSWORD": "4eff2199-185d-4919-b335-b5c339e731f0", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 200 OK in 100.923µs >[heketi] INFO 2018/08/19 14:11:20 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:11:20 asynchttp.go:292: Completed job 4ca8dfec34ba07abb49a78c4557403ef in 8.021183095s >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 135.652µs >[kubeexec] DEBUG 2018/08/19 14:11:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:11:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/4ca8dfec34ba07abb49a78c4557403ef >[negroni] Completed 303 See Other in 202.183µs >[negroni] Started GET /blockvolumes/d2d79dee195dcaaa113173558896208d >[negroni] Completed 200 OK in 1.575621ms >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 119.496µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 173.377µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 122.212µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 166.337µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 130.496µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 163.415µs >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 200 OK in 125.861µs >[kubeexec] DEBUG 2018/08/19 14:11:28 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_9dd7a04556063495f445881a7cd5d68f ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:73ac4cc2-0e33-4a33-8375-9519c11fec4e", "USERNAME": "73ac4cc2-0e33-4a33-8375-9519c11fec4e", "PASSWORD": "79fe2e35-abd0-4920-b659-e96bf48a00c9", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:11:28 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:11:28 asynchttp.go:292: Completed job 4f8b27df8dbde661d37a3bbcdb6de313 in 14.642940198s >[negroni] Started GET /queue/4f8b27df8dbde661d37a3bbcdb6de313 >[negroni] Completed 303 See Other in 128.026µs >[negroni] Started GET /blockvolumes/9dd7a04556063495f445881a7cd5d68f >[negroni] Completed 200 OK in 2.838914ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:11:43 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 42.885924ms >[asynchttp] INFO 2018/08/19 14:11:43 asynchttp.go:288: Started job 1c8a84ecc3253f641eb013b01eb36fdb >[heketi] INFO 2018/08/19 14:11:43 Started async operation: Create Block Volume >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 114.129µs >[cmdexec] INFO 2018/08/19 14:11:43 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 9min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:11:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:11:43 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 15min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:11:43 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 120.944µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:11:44 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 46.493648ms >[asynchttp] INFO 2018/08/19 14:11:44 asynchttp.go:288: Started job 67838b986f39b38d406a8c11f5cc8b9f >[heketi] INFO 2018/08/19 14:11:44 Started async operation: Create Block Volume >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 114.232µs >[cmdexec] INFO 2018/08/19 14:11:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 146.876µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 180.159µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 166.625µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 158.142µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 121.24µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 169.21µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 172.613µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 149.621µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 159.186µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 162.673µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 175.048µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 119.227µs >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 200 OK in 124.033µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 123.288µs >[kubeexec] DEBUG 2018/08/19 14:11:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f0a8270410134a8952a0cd8ae523e15f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:5ad8aa79-de90-42cf-93b1-4708a77f719d", "USERNAME": "5ad8aa79-de90-42cf-93b1-4708a77f719d", "PASSWORD": "9502a82c-3ce8-43ea-9885-09014748d21e", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:11:51 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:11:51 asynchttp.go:292: Completed job 1c8a84ecc3253f641eb013b01eb36fdb in 8.35643s >[kubeexec] DEBUG 2018/08/19 14:11:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:11:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 9min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:11:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 15min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:11:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:11:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 15min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1c8a84ecc3253f641eb013b01eb36fdb >[negroni] Completed 303 See Other in 134.242µs >[negroni] Started GET /blockvolumes/f0a8270410134a8952a0cd8ae523e15f >[negroni] Completed 200 OK in 1.550672ms >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 121.021µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 139.738µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 161.187µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 189.812µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 269.308µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 123.673µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 120.603µs >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 200 OK in 162.608µs >[kubeexec] DEBUG 2018/08/19 14:11:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_79e6508d4b8405690e4e690268a71f67 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:e747f839-9157-4229-9081-c14902e8d6e9", "USERNAME": "e747f839-9157-4229-9081-c14902e8d6e9", "PASSWORD": "fea6f5e1-b246-447e-abab-636ec7e567bf", "PORTAL(S)": [ "10.70.47.183:3260", "10.70.47.54:3260", "10.70.46.152:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:11:59 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:11:59 asynchttp.go:292: Completed job 67838b986f39b38d406a8c11f5cc8b9f in 15.073711706s >[negroni] Started GET /queue/67838b986f39b38d406a8c11f5cc8b9f >[negroni] Completed 303 See Other in 129.416µs >[negroni] Started GET /blockvolumes/79e6508d4b8405690e4e690268a71f67 >[negroni] Completed 200 OK in 2.184386ms >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:12:13 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 42.538475ms >[asynchttp] INFO 2018/08/19 14:12:13 asynchttp.go:288: Started job c420b4c146227dc9fa8d8b2fff801d64 >[heketi] INFO 2018/08/19 14:12:13 Started async operation: Create Block Volume >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 174.223µs >[cmdexec] INFO 2018/08/19 14:12:13 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 15min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:12:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:12:13 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 9min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:12:13 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:13 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 156.998µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:12:14 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 44.025031ms >[asynchttp] INFO 2018/08/19 14:12:14 asynchttp.go:288: Started job a46a243cd3d74eb4604e2347fe0439da >[heketi] INFO 2018/08/19 14:12:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 108.766µs >[cmdexec] INFO 2018/08/19 14:12:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 15min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:12:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 124.758µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 156.649µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 128.621µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 171.473µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 125.958µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 120.514µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 177.508µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 117.212µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 164.166µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 137.875µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 123.83µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 126.69µs >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 200 OK in 124.026µs >[kubeexec] DEBUG 2018/08/19 14:12:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5d3fa67df8c65eacc505c7b6c33fcd45 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:fb991dc0-f0f7-42d2-9080-2ec76f7b6b64", "USERNAME": "fb991dc0-f0f7-42d2-9080-2ec76f7b6b64", "PASSWORD": "decc2ff2-64e9-44cf-ba79-7666ff6bf095", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 155.775µs >[heketi] INFO 2018/08/19 14:12:21 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:12:21 asynchttp.go:292: Completed job c420b4c146227dc9fa8d8b2fff801d64 in 8.205819531s >[kubeexec] DEBUG 2018/08/19 14:12:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:12:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 9min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:12:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c420b4c146227dc9fa8d8b2fff801d64 >[negroni] Completed 303 See Other in 178.563µs >[negroni] Started GET /blockvolumes/5d3fa67df8c65eacc505c7b6c33fcd45 >[negroni] Completed 200 OK in 1.53266ms >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 127.561µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 123.997µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 141.199µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 173.637µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 154.614µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 138.381µs >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 200 OK in 135.65µs >[kubeexec] DEBUG 2018/08/19 14:12:29 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_777535468254d40d930bab8cc1522c6b ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:52f748ad-ba79-4657-b86f-edeede001a41", "USERNAME": "52f748ad-ba79-4657-b86f-edeede001a41", "PASSWORD": "fd88b79b-a61e-4e2f-8bc4-2b38600dd562", "PORTAL(S)": [ "10.70.46.152:3260", "10.70.47.183:3260", "10.70.47.54:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:12:29 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:12:29 asynchttp.go:292: Completed job a46a243cd3d74eb4604e2347fe0439da in 15.006732224s >[negroni] Started GET /queue/a46a243cd3d74eb4604e2347fe0439da >[negroni] Completed 303 See Other in 105.424µs >[negroni] Started GET /blockvolumes/777535468254d40d930bab8cc1522c6b >[negroni] Completed 200 OK in 2.578463ms >[heketi] INFO 2018/08/19 14:12:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:12:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:12:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:12:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:12:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:12:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:12:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:12:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:12:43 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.664112ms >[asynchttp] INFO 2018/08/19 14:12:43 asynchttp.go:288: Started job 403acd82f0f8213491aee8caae10e362 >[heketi] INFO 2018/08/19 14:12:43 Started async operation: Create Block Volume >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 154.153µs >[cmdexec] INFO 2018/08/19 14:12:43 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:43 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:12:43 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:12:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:12:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 143.452µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:12:44 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 38.234693ms >[asynchttp] INFO 2018/08/19 14:12:44 asynchttp.go:288: Started job d1a92ff5d998cbc5deaa285f85e9d205 >[heketi] INFO 2018/08/19 14:12:44 Started async operation: Create Block Volume >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 121.532µs >[cmdexec] INFO 2018/08/19 14:12:44 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:12:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 123.189µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 142.939µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 188.507µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 119.42µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 120.412µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 136.482µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 176.456µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 152.212µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 128.438µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 174.342µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 140.434µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 127.183µs >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 200 OK in 130.996µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 127.401µs >[kubeexec] DEBUG 2018/08/19 14:12:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d7400ff34c5f3205d616ca782a9a1cd6 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json >Result: { "IQN": "iqn.2016-12.org.gluster-block:a1fadbea-2e28-41ea-82eb-388519681235", "USERNAME": "a1fadbea-2e28-41ea-82eb-388519681235", "PASSWORD": "ad2a7746-d221-4d0a-8662-4e7eeb427a90", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.46.152:3260", "10.70.47.183:3260" ], "RESULT": "SUCCESS" } >[heketi] INFO 2018/08/19 14:12:52 Create Block Volume succeeded >[asynchttp] INFO 2018/08/19 14:12:52 asynchttp.go:292: Completed job 403acd82f0f8213491aee8caae10e362 in 8.352973713s >[kubeexec] DEBUG 2018/08/19 14:12:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:12:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:12:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:12:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/403acd82f0f8213491aee8caae10e362 >[negroni] Completed 303 See Other in 198.124µs >[negroni] Started GET /blockvolumes/d7400ff34c5f3205d616ca782a9a1cd6 >[negroni] Completed 200 OK in 1.731809ms >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 181.54µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 159.515µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 125.606µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 210.819µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 135.207µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 147.139µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 175.7µs >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 200 OK in 148.923µs >[kubeexec] ERROR 2018/08/19 14:12:59 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_39dcf20c41658286956878571c2cfce6 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "IQN": "iqn.2016-12.org.gluster-block:3e6c8b39-09c5-484a-b7ce-a6bfea7192f2", "USERNAME": "3e6c8b39-09c5-484a-b7ce-a6bfea7192f2", "PASSWORD": "d5d5f7d2-1d14-40e4-8c29-fa6de3bb2bb4", "PORTAL(S)": [ "10.70.47.54:3260", "10.70.47.183:3260", "10.70.46.152:3260" ], "ROLLBACK FAILED ON": [ "10.70.47.54", "10.70.47.183", "10.70.46.152" ], "RESULT": "FAIL" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_39dcf20c41658286956878571c2cfce6 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_39dcf20c41658286956878571c2cfce6: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:13:00 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 16min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_39dcf20c41658286956878571c2cfce6 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_39dcf20c41658286956878571c2cfce6: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:00 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:13:00 asynchttp.go:292: Completed job d1a92ff5d998cbc5deaa285f85e9d205 in 15.646095436s >[negroni] Started GET /queue/d1a92ff5d998cbc5deaa285f85e9d205 >[negroni] Completed 500 Internal Server Error in 137.945µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:13:14 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 44.792291ms >[asynchttp] INFO 2018/08/19 14:13:14 asynchttp.go:288: Started job 3d2bdd8c835465b733002577fa49f7f4 >[heketi] INFO 2018/08/19 14:13:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/3d2bdd8c835465b733002577fa49f7f4 >[negroni] Completed 200 OK in 115.547µs >[cmdexec] INFO 2018/08/19 14:13:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:13:14 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:13:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:13:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:13:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_9e2787f26fd542263cb4ff1a106afbc7 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_9e2787f26fd542263cb4ff1a106afbc7[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_9e2787f26fd542263cb4ff1a106afbc7 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_9e2787f26fd542263cb4ff1a106afbc7: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:13:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3d2bdd8c835465b733002577fa49f7f4 >[negroni] Completed 200 OK in 127.734µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:13:15 Free size 0 is less than the requested block volume size 1 >[kubeexec] DEBUG 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Completed 202 Accepted in 48.749906ms >[asynchttp] INFO 2018/08/19 14:13:15 asynchttp.go:288: Started job 48461959f22e9d106d51de8ae145f7c8 >[heketi] INFO 2018/08/19 14:13:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/48461959f22e9d106d51de8ae145f7c8 >[negroni] Completed 200 OK in 121.222µs >[cmdexec] INFO 2018/08/19 14:13:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_9e2787f26fd542263cb4ff1a106afbc7 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_9e2787f26fd542263cb4ff1a106afbc7: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:15 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:13:15 asynchttp.go:292: Completed job 3d2bdd8c835465b733002577fa49f7f4 in 1.284309136s >[kubeexec] DEBUG 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 10min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:13:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 16min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:13:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3d2bdd8c835465b733002577fa49f7f4 >[negroni] Completed 500 Internal Server Error in 168.703µs >[negroni] Started GET /queue/48461959f22e9d106d51de8ae145f7c8 >[negroni] Completed 200 OK in 124.276µs >[kubeexec] DEBUG 2018/08/19 14:13:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:13:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:13:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_dacaa60233967de8bc305d38cc2f4127 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_dacaa60233967de8bc305d38cc2f4127[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/48461959f22e9d106d51de8ae145f7c8 >[negroni] Completed 200 OK in 135.614µs >[kubeexec] ERROR 2018/08/19 14:13:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_dacaa60233967de8bc305d38cc2f4127 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:17 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_dacaa60233967de8bc305d38cc2f4127: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:17 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:13:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/48461959f22e9d106d51de8ae145f7c8 >[negroni] Completed 200 OK in 166.908µs >[kubeexec] ERROR 2018/08/19 14:13:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_dacaa60233967de8bc305d38cc2f4127 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:18 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_dacaa60233967de8bc305d38cc2f4127: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:18 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:18 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:13:18 asynchttp.go:292: Completed job 48461959f22e9d106d51de8ae145f7c8 in 3.249871836s >[negroni] Started GET /queue/48461959f22e9d106d51de8ae145f7c8 >[negroni] Completed 500 Internal Server Error in 136.708µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:13:44 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 54.236949ms >[asynchttp] INFO 2018/08/19 14:13:44 asynchttp.go:288: Started job a914b87958c4597bbe29dca8bd132775 >[heketi] INFO 2018/08/19 14:13:44 Started async operation: Create Block Volume >[negroni] Started GET /queue/a914b87958c4597bbe29dca8bd132775 >[negroni] Completed 200 OK in 132.212µs >[cmdexec] INFO 2018/08/19 14:13:44 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 17min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:13:44 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 11min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:13:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:13:44 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:44 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e60d4424834ec798f94caf5b383299e3 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_e60d4424834ec798f94caf5b383299e3[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e60d4424834ec798f94caf5b383299e3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e60d4424834ec798f94caf5b383299e3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:13:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/a914b87958c4597bbe29dca8bd132775 >[negroni] Completed 200 OK in 134.418µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:13:45 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 46.586831ms >[asynchttp] INFO 2018/08/19 14:13:45 asynchttp.go:288: Started job 6c94b96f7e0b0daf9730f0e40da600b5 >[heketi] INFO 2018/08/19 14:13:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/6c94b96f7e0b0daf9730f0e40da600b5 >[negroni] Completed 200 OK in 153.525µs >[cmdexec] INFO 2018/08/19 14:13:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e60d4424834ec798f94caf5b383299e3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e60d4424834ec798f94caf5b383299e3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:45 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:13:45 asynchttp.go:292: Completed job a914b87958c4597bbe29dca8bd132775 in 1.243137184s >[kubeexec] DEBUG 2018/08/19 14:13:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:13:45 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 11min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:13:46 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/a914b87958c4597bbe29dca8bd132775 >[negroni] Completed 500 Internal Server Error in 142.112µs >[negroni] Started GET /queue/6c94b96f7e0b0daf9730f0e40da600b5 >[negroni] Completed 200 OK in 120.536µs >[kubeexec] DEBUG 2018/08/19 14:13:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 17min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:13:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6c94b96f7e0b0daf9730f0e40da600b5 >[negroni] Completed 200 OK in 126.633µs >[kubeexec] ERROR 2018/08/19 14:13:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5029a47b5c267f6d8d71f34055aa427c ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_5029a47b5c267f6d8d71f34055aa427c[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_5029a47b5c267f6d8d71f34055aa427c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_5029a47b5c267f6d8d71f34055aa427c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:13:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:13:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 17min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6c94b96f7e0b0daf9730f0e40da600b5 >[negroni] Completed 200 OK in 118.736µs >[kubeexec] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_5029a47b5c267f6d8d71f34055aa427c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_5029a47b5c267f6d8d71f34055aa427c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:13:48 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:13:48 asynchttp.go:292: Completed job 6c94b96f7e0b0daf9730f0e40da600b5 in 3.279995782s >[negroni] Started GET /queue/6c94b96f7e0b0daf9730f0e40da600b5 >[negroni] Completed 500 Internal Server Error in 158.082µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:14:14 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.155173ms >[asynchttp] INFO 2018/08/19 14:14:14 asynchttp.go:288: Started job c2aaa8eca2b73dfbcdbfc0f4f09670dc >[heketi] INFO 2018/08/19 14:14:14 Started async operation: Create Block Volume >[negroni] Started GET /queue/c2aaa8eca2b73dfbcdbfc0f4f09670dc >[negroni] Completed 200 OK in 154.148µs >[cmdexec] INFO 2018/08/19 14:14:14 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:14 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:14:14 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 17min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:14:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 11min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:14:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d6d486c2548633ebc2f6f55b0a274471 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_d6d486c2548633ebc2f6f55b0a274471[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d6d486c2548633ebc2f6f55b0a274471 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d6d486c2548633ebc2f6f55b0a274471: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:14:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c2aaa8eca2b73dfbcdbfc0f4f09670dc >[negroni] Completed 200 OK in 134.349µs >[kubeexec] DEBUG 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:14:15 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.63207ms >[asynchttp] INFO 2018/08/19 14:14:15 asynchttp.go:288: Started job 8b59f84ebb271a6f85ef6e5dbf7d6f38 >[heketi] INFO 2018/08/19 14:14:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/8b59f84ebb271a6f85ef6e5dbf7d6f38 >[negroni] Completed 200 OK in 111.543µs >[cmdexec] INFO 2018/08/19 14:14:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d6d486c2548633ebc2f6f55b0a274471 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d6d486c2548633ebc2f6f55b0a274471: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:15 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:14:16 asynchttp.go:292: Completed job c2aaa8eca2b73dfbcdbfc0f4f09670dc in 1.241253473s >[heketi] ERROR 2018/08/19 14:14:16 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:14:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 17min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:14:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:14:16 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c2aaa8eca2b73dfbcdbfc0f4f09670dc >[negroni] Completed 500 Internal Server Error in 110.261µs >[negroni] Started GET /queue/8b59f84ebb271a6f85ef6e5dbf7d6f38 >[negroni] Completed 200 OK in 159.204µs >[kubeexec] DEBUG 2018/08/19 14:14:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 11min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:14:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8b59f84ebb271a6f85ef6e5dbf7d6f38 >[negroni] Completed 200 OK in 201.532µs >[kubeexec] ERROR 2018/08/19 14:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0866259496613677879d33131b264931 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_0866259496613677879d33131b264931[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_0866259496613677879d33131b264931 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:18 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_0866259496613677879d33131b264931: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:18 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:14:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/8b59f84ebb271a6f85ef6e5dbf7d6f38 >[negroni] Completed 200 OK in 122.378µs >[kubeexec] ERROR 2018/08/19 14:14:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_0866259496613677879d33131b264931 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_0866259496613677879d33131b264931: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:19 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:19 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:14:19 asynchttp.go:292: Completed job 8b59f84ebb271a6f85ef6e5dbf7d6f38 in 3.330901216s >[negroni] Started GET /queue/8b59f84ebb271a6f85ef6e5dbf7d6f38 >[negroni] Completed 500 Internal Server Error in 157.178µs >[heketi] INFO 2018/08/19 14:14:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:14:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:14:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:14:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:14:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:14:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:14:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:14:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:14:45 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 47.578318ms >[asynchttp] INFO 2018/08/19 14:14:45 asynchttp.go:288: Started job f6aad2f8352b3746ec091ef88c25b962 >[heketi] INFO 2018/08/19 14:14:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/f6aad2f8352b3746ec091ef88c25b962 >[negroni] Completed 200 OK in 144.507µs >[cmdexec] INFO 2018/08/19 14:14:45 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:14:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:14:45 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:14:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_81e7a95ca3af1195591a2bb9a000d68c ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_81e7a95ca3af1195591a2bb9a000d68c[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_81e7a95ca3af1195591a2bb9a000d68c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:45 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_81e7a95ca3af1195591a2bb9a000d68c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:45 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:14:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f6aad2f8352b3746ec091ef88c25b962 >[negroni] Completed 200 OK in 137.083µs >[kubeexec] DEBUG 2018/08/19 14:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:14:46 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 53.543967ms >[asynchttp] INFO 2018/08/19 14:14:46 asynchttp.go:288: Started job 815d672a0a6bb2f7dfd7226c46c61e99 >[heketi] INFO 2018/08/19 14:14:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/815d672a0a6bb2f7dfd7226c46c61e99 >[negroni] Completed 200 OK in 153.948µs >[cmdexec] INFO 2018/08/19 14:14:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_81e7a95ca3af1195591a2bb9a000d68c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:46 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_81e7a95ca3af1195591a2bb9a000d68c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:46 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:46 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:14:46 asynchttp.go:292: Completed job f6aad2f8352b3746ec091ef88c25b962 in 1.34169166s >[kubeexec] DEBUG 2018/08/19 14:14:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:14:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f6aad2f8352b3746ec091ef88c25b962 >[negroni] Completed 500 Internal Server Error in 119.14µs >[kubeexec] DEBUG 2018/08/19 14:14:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:14:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/815d672a0a6bb2f7dfd7226c46c61e99 >[negroni] Completed 200 OK in 100.993µs >[kubeexec] DEBUG 2018/08/19 14:14:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:14:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/815d672a0a6bb2f7dfd7226c46c61e99 >[negroni] Completed 200 OK in 142.754µs >[kubeexec] ERROR 2018/08/19 14:14:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_8c81d5a5c573897e9bfc392ff139f094 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_8c81d5a5c573897e9bfc392ff139f094[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:14:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_8c81d5a5c573897e9bfc392ff139f094 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_8c81d5a5c573897e9bfc392ff139f094: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:14:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:14:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 18min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/815d672a0a6bb2f7dfd7226c46c61e99 >[negroni] Completed 200 OK in 215.797µs >[kubeexec] ERROR 2018/08/19 14:14:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_8c81d5a5c573897e9bfc392ff139f094 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:14:49 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_8c81d5a5c573897e9bfc392ff139f094: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:14:49 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:14:49 asynchttp.go:292: Completed job 815d672a0a6bb2f7dfd7226c46c61e99 in 3.342992138s >[heketi] ERROR 2018/08/19 14:14:49 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/815d672a0a6bb2f7dfd7226c46c61e99 >[negroni] Completed 500 Internal Server Error in 133.753µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:15:15 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 96.797938ms >[asynchttp] INFO 2018/08/19 14:15:15 asynchttp.go:288: Started job c0236f5c0aa86489ec1e13b17aaee54e >[heketi] INFO 2018/08/19 14:15:15 Started async operation: Create Block Volume >[negroni] Started GET /queue/c0236f5c0aa86489ec1e13b17aaee54e >[negroni] Completed 200 OK in 122.215µs >[cmdexec] INFO 2018/08/19 14:15:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:15:15 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:15:15 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:15 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:15:15 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_825fd6f41e04b2e134415d85abd109e0 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_825fd6f41e04b2e134415d85abd109e0[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_825fd6f41e04b2e134415d85abd109e0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_825fd6f41e04b2e134415d85abd109e0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:15:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/c0236f5c0aa86489ec1e13b17aaee54e >[negroni] Completed 200 OK in 108.386µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:15:16 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 49.161235ms >[asynchttp] INFO 2018/08/19 14:15:16 asynchttp.go:288: Started job b45790d4596572ae6eab95fb63acdd28 >[heketi] INFO 2018/08/19 14:15:16 Started async operation: Create Block Volume >[negroni] Started GET /queue/b45790d4596572ae6eab95fb63acdd28 >[negroni] Completed 200 OK in 167.438µs >[cmdexec] INFO 2018/08/19 14:15:16 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_825fd6f41e04b2e134415d85abd109e0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_825fd6f41e04b2e134415d85abd109e0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:15:16 asynchttp.go:292: Completed job c0236f5c0aa86489ec1e13b17aaee54e in 1.324139177s >[heketi] ERROR 2018/08/19 14:15:16 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:15:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 12min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:15:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/c0236f5c0aa86489ec1e13b17aaee54e >[negroni] Completed 500 Internal Server Error in 133.793µs >[kubeexec] DEBUG 2018/08/19 14:15:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 18min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:15:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/b45790d4596572ae6eab95fb63acdd28 >[negroni] Completed 200 OK in 113.484µs >[kubeexec] DEBUG 2018/08/19 14:15:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:15:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b45790d4596572ae6eab95fb63acdd28 >[negroni] Completed 200 OK in 167.421µs >[kubeexec] ERROR 2018/08/19 14:15:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e6ea4b096715795da0ae5d28e18f7c2a ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_e6ea4b096715795da0ae5d28e18f7c2a[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:15:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e6ea4b096715795da0ae5d28e18f7c2a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e6ea4b096715795da0ae5d28e18f7c2a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:15:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b45790d4596572ae6eab95fb63acdd28 >[negroni] Completed 200 OK in 107.321µs >[kubeexec] ERROR 2018/08/19 14:15:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e6ea4b096715795da0ae5d28e18f7c2a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:20 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e6ea4b096715795da0ae5d28e18f7c2a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:20 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:15:20 asynchttp.go:292: Completed job b45790d4596572ae6eab95fb63acdd28 in 3.54864304s >[heketi] ERROR 2018/08/19 14:15:20 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/b45790d4596572ae6eab95fb63acdd28 >[negroni] Completed 500 Internal Server Error in 162.886µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:15:45 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 47.61897ms >[asynchttp] INFO 2018/08/19 14:15:45 asynchttp.go:288: Started job fbcefba5e847c3e55df674e94e2c3d1a >[heketi] INFO 2018/08/19 14:15:45 Started async operation: Create Block Volume >[negroni] Started GET /queue/fbcefba5e847c3e55df674e94e2c3d1a >[negroni] Completed 200 OK in 100.618µs >[cmdexec] INFO 2018/08/19 14:15:45 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:45 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:15:45 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 19min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:15:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 13min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:15:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_87279addba5b190838380c20efb86ef0 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_87279addba5b190838380c20efb86ef0[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_87279addba5b190838380c20efb86ef0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_87279addba5b190838380c20efb86ef0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:46 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:15:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fbcefba5e847c3e55df674e94e2c3d1a >[negroni] Completed 200 OK in 128.196µs >[kubeexec] DEBUG 2018/08/19 14:15:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:15:46 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.028489ms >[asynchttp] INFO 2018/08/19 14:15:46 asynchttp.go:288: Started job 276d93894adb868513500382e090d189 >[heketi] INFO 2018/08/19 14:15:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/276d93894adb868513500382e090d189 >[negroni] Completed 200 OK in 114.469µs >[cmdexec] INFO 2018/08/19 14:15:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:15:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_87279addba5b190838380c20efb86ef0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:47 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_87279addba5b190838380c20efb86ef0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:47 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:15:47 asynchttp.go:292: Completed job fbcefba5e847c3e55df674e94e2c3d1a in 1.302789929s >[heketi] ERROR 2018/08/19 14:15:47 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:15:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:15:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 13min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:15:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fbcefba5e847c3e55df674e94e2c3d1a >[negroni] Completed 500 Internal Server Error in 136.57µs >[negroni] Started GET /queue/276d93894adb868513500382e090d189 >[negroni] Completed 200 OK in 174.815µs >[kubeexec] DEBUG 2018/08/19 14:15:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 19min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:15:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/276d93894adb868513500382e090d189 >[negroni] Completed 200 OK in 129.026µs >[kubeexec] ERROR 2018/08/19 14:15:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_3701c0b6e444b23aa095852453b6714c ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_3701c0b6e444b23aa095852453b6714c[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:15:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_3701c0b6e444b23aa095852453b6714c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:49 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_3701c0b6e444b23aa095852453b6714c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:49 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:15:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:15:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 19min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/276d93894adb868513500382e090d189 >[negroni] Completed 200 OK in 128.217µs >[kubeexec] ERROR 2018/08/19 14:15:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_3701c0b6e444b23aa095852453b6714c --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:15:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_3701c0b6e444b23aa095852453b6714c: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:50 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:15:50 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:15:50 asynchttp.go:292: Completed job 276d93894adb868513500382e090d189 in 3.26706946s >[negroni] Started GET /queue/276d93894adb868513500382e090d189 >[negroni] Completed 500 Internal Server Error in 124.104µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:16:16 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 46.828905ms >[asynchttp] INFO 2018/08/19 14:16:16 asynchttp.go:288: Started job afe5bfb248d9b7149b1c7135c951f825 >[heketi] INFO 2018/08/19 14:16:16 Started async operation: Create Block Volume >[negroni] Started GET /queue/afe5bfb248d9b7149b1c7135c951f825 >[negroni] Completed 200 OK in 101.793µs >[cmdexec] INFO 2018/08/19 14:16:16 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 13min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:16:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:16:16 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 19min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:16:16 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:16:16 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:16:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/afe5bfb248d9b7149b1c7135c951f825 >[negroni] Completed 200 OK in 173.809µs >[kubeexec] DEBUG 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:16:17 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 162.515146ms >[asynchttp] INFO 2018/08/19 14:16:17 asynchttp.go:288: Started job 80272b6d23b10b9c8be4c342bda095c4 >[heketi] INFO 2018/08/19 14:16:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/80272b6d23b10b9c8be4c342bda095c4 >[negroni] Completed 200 OK in 152.872µs >[cmdexec] INFO 2018/08/19 14:16:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_20da0a9e6f2a0faba6c7ea4d8fc1734f: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:17 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:16:17 asynchttp.go:292: Completed job afe5bfb248d9b7149b1c7135c951f825 in 1.382062029s >[kubeexec] DEBUG 2018/08/19 14:16:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 19min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:16:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/afe5bfb248d9b7149b1c7135c951f825 >[negroni] Completed 500 Internal Server Error in 122.759µs >[kubeexec] DEBUG 2018/08/19 14:16:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:16:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/80272b6d23b10b9c8be4c342bda095c4 >[negroni] Completed 200 OK in 165.62µs >[kubeexec] DEBUG 2018/08/19 14:16:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 13min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:16:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:16:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/80272b6d23b10b9c8be4c342bda095c4 >[negroni] Completed 200 OK in 141.364µs >[kubeexec] ERROR 2018/08/19 14:16:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:16:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/80272b6d23b10b9c8be4c342bda095c4 >[negroni] Completed 200 OK in 117.886µs >[kubeexec] ERROR 2018/08/19 14:16:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:20 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_2270ad4ecc7a59ca0ef7ecba6c29b2c3: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:20 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:20 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:16:20 asynchttp.go:292: Completed job 80272b6d23b10b9c8be4c342bda095c4 in 3.223126806s >[negroni] Started GET /queue/80272b6d23b10b9c8be4c342bda095c4 >[negroni] Completed 500 Internal Server Error in 147.236µs >[heketi] INFO 2018/08/19 14:16:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:16:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:16:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:16:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:16:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:16:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:16:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:16:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:16:46 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 49.417642ms >[asynchttp] INFO 2018/08/19 14:16:46 asynchttp.go:288: Started job 93b4a73800e4ffee327eb8af2789d3ea >[heketi] INFO 2018/08/19 14:16:46 Started async operation: Create Block Volume >[negroni] Started GET /queue/93b4a73800e4ffee327eb8af2789d3ea >[negroni] Completed 200 OK in 116.699µs >[cmdexec] INFO 2018/08/19 14:16:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:16:46 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:16:46 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:46 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:16:46 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e4af43280ea196a7f6e9838e82fdf9ac ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_e4af43280ea196a7f6e9838e82fdf9ac[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e4af43280ea196a7f6e9838e82fdf9ac --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e4af43280ea196a7f6e9838e82fdf9ac: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:16:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/93b4a73800e4ffee327eb8af2789d3ea >[negroni] Completed 200 OK in 125.147µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:16:47 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.522895ms >[asynchttp] INFO 2018/08/19 14:16:47 asynchttp.go:288: Started job 1f306a49ad23ed52ddf950c16c27d467 >[heketi] INFO 2018/08/19 14:16:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/1f306a49ad23ed52ddf950c16c27d467 >[negroni] Completed 200 OK in 122.691µs >[cmdexec] INFO 2018/08/19 14:16:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e4af43280ea196a7f6e9838e82fdf9ac --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e4af43280ea196a7f6e9838e82fdf9ac: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:47 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:16:47 asynchttp.go:292: Completed job 93b4a73800e4ffee327eb8af2789d3ea in 1.313806596s >[kubeexec] DEBUG 2018/08/19 14:16:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:16:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/93b4a73800e4ffee327eb8af2789d3ea >[negroni] Completed 500 Internal Server Error in 122.583µs >[kubeexec] DEBUG 2018/08/19 14:16:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:16:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/1f306a49ad23ed52ddf950c16c27d467 >[negroni] Completed 200 OK in 174.185µs >[kubeexec] DEBUG 2018/08/19 14:16:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:16:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1f306a49ad23ed52ddf950c16c27d467 >[negroni] Completed 200 OK in 125.44µs >[kubeexec] ERROR 2018/08/19 14:16:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_450103bd86bfb3dacc936cf1294a84f8 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_450103bd86bfb3dacc936cf1294a84f8[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_450103bd86bfb3dacc936cf1294a84f8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_450103bd86bfb3dacc936cf1294a84f8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:16:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:16:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 20min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/1f306a49ad23ed52ddf950c16c27d467 >[negroni] Completed 200 OK in 176.263µs >[kubeexec] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_450103bd86bfb3dacc936cf1294a84f8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_450103bd86bfb3dacc936cf1294a84f8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:50 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:16:51 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:16:51 asynchttp.go:292: Completed job 1f306a49ad23ed52ddf950c16c27d467 in 3.424343916s >[negroni] Started GET /queue/1f306a49ad23ed52ddf950c16c27d467 >[negroni] Completed 500 Internal Server Error in 127.413µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:17:16 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 42.450282ms >[asynchttp] INFO 2018/08/19 14:17:16 asynchttp.go:288: Started job 4f5c2fd657ec07244686a3ab83166946 >[heketi] INFO 2018/08/19 14:17:16 Started async operation: Create Block Volume >[negroni] Started GET /queue/4f5c2fd657ec07244686a3ab83166946 >[negroni] Completed 200 OK in 123.113µs >[cmdexec] INFO 2018/08/19 14:17:16 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 20min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:17:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:17:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:17:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_802a4542402de26914124e1b97d8bf9e ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_802a4542402de26914124e1b97d8bf9e[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/4f5c2fd657ec07244686a3ab83166946 >[negroni] Completed 200 OK in 177.509µs >[kubeexec] ERROR 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_802a4542402de26914124e1b97d8bf9e --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:17 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_802a4542402de26914124e1b97d8bf9e: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:17 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:17:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:17:17 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 53.094311ms >[asynchttp] INFO 2018/08/19 14:17:18 asynchttp.go:288: Started job 66beae2be790126163c27f10e10f5af4 >[heketi] INFO 2018/08/19 14:17:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/66beae2be790126163c27f10e10f5af4 >[negroni] Completed 200 OK in 119.268µs >[cmdexec] INFO 2018/08/19 14:17:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] DEBUG 2018/08/19 14:17:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 14min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:17:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:17:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_802a4542402de26914124e1b97d8bf9e --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:18 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_802a4542402de26914124e1b97d8bf9e: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:18 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:17:18 asynchttp.go:292: Completed job 4f5c2fd657ec07244686a3ab83166946 in 1.704379231s >[heketi] ERROR 2018/08/19 14:17:18 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:17:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:17:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4f5c2fd657ec07244686a3ab83166946 >[negroni] Completed 500 Internal Server Error in 160.047µs >[negroni] Started GET /queue/66beae2be790126163c27f10e10f5af4 >[negroni] Completed 200 OK in 155.455µs >[kubeexec] DEBUG 2018/08/19 14:17:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 21min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:17:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:17:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_81e08213fe38e6b915ccaeb016aaff82 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_81e08213fe38e6b915ccaeb016aaff82[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/66beae2be790126163c27f10e10f5af4 >[negroni] Completed 200 OK in 147.465µs >[kubeexec] ERROR 2018/08/19 14:17:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_81e08213fe38e6b915ccaeb016aaff82 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:20 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_81e08213fe38e6b915ccaeb016aaff82: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:20 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:17:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/66beae2be790126163c27f10e10f5af4 >[negroni] Completed 200 OK in 146.474µs >[kubeexec] ERROR 2018/08/19 14:17:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_81e08213fe38e6b915ccaeb016aaff82 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_81e08213fe38e6b915ccaeb016aaff82: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:21 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:17:21 asynchttp.go:292: Completed job 66beae2be790126163c27f10e10f5af4 in 3.231266069s >[heketi] ERROR 2018/08/19 14:17:21 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/66beae2be790126163c27f10e10f5af4 >[negroni] Completed 500 Internal Server Error in 127.115µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:17:47 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.226761ms >[asynchttp] INFO 2018/08/19 14:17:47 asynchttp.go:288: Started job 27d6a5cafcf4fd0ee61b03ca71a1d58d >[heketi] INFO 2018/08/19 14:17:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/27d6a5cafcf4fd0ee61b03ca71a1d58d >[negroni] Completed 200 OK in 141.953µs >[cmdexec] INFO 2018/08/19 14:17:47 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 21min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:17:47 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 15min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:17:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:17:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:17:47 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_500485782ebda0fdffa05fbea3c53da2 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_500485782ebda0fdffa05fbea3c53da2[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_500485782ebda0fdffa05fbea3c53da2 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_500485782ebda0fdffa05fbea3c53da2: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:17:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/27d6a5cafcf4fd0ee61b03ca71a1d58d >[negroni] Completed 200 OK in 129.728µs >[kubeexec] DEBUG 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:17:48 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 46.791483ms >[asynchttp] INFO 2018/08/19 14:17:48 asynchttp.go:288: Started job ab81f54b35ad8a91489e3e7c2f8c9df0 >[heketi] INFO 2018/08/19 14:17:48 Started async operation: Create Block Volume >[negroni] Started GET /queue/ab81f54b35ad8a91489e3e7c2f8c9df0 >[negroni] Completed 200 OK in 127.087µs >[cmdexec] INFO 2018/08/19 14:17:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_500485782ebda0fdffa05fbea3c53da2 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_500485782ebda0fdffa05fbea3c53da2: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:48 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:17:48 asynchttp.go:292: Completed job 27d6a5cafcf4fd0ee61b03ca71a1d58d in 1.300503594s >[kubeexec] DEBUG 2018/08/19 14:17:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 21min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:17:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/27d6a5cafcf4fd0ee61b03ca71a1d58d >[negroni] Completed 500 Internal Server Error in 171.173µs >[kubeexec] DEBUG 2018/08/19 14:17:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:17:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ab81f54b35ad8a91489e3e7c2f8c9df0 >[negroni] Completed 200 OK in 188.271µs >[kubeexec] DEBUG 2018/08/19 14:17:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 15min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:17:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ab81f54b35ad8a91489e3e7c2f8c9df0 >[negroni] Completed 200 OK in 127.388µs >[kubeexec] ERROR 2018/08/19 14:17:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_07d566097b87535f30d6c37dde268780 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_07d566097b87535f30d6c37dde268780[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:17:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_07d566097b87535f30d6c37dde268780 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_07d566097b87535f30d6c37dde268780: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:17:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:17:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 21min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ab81f54b35ad8a91489e3e7c2f8c9df0 >[negroni] Completed 200 OK in 157.789µs >[kubeexec] ERROR 2018/08/19 14:17:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_07d566097b87535f30d6c37dde268780 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:17:51 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_07d566097b87535f30d6c37dde268780: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:51 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:17:51 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:17:51 asynchttp.go:292: Completed job ab81f54b35ad8a91489e3e7c2f8c9df0 in 3.323167184s >[negroni] Started GET /queue/ab81f54b35ad8a91489e3e7c2f8c9df0 >[negroni] Completed 500 Internal Server Error in 126.777µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:18:17 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.485477ms >[asynchttp] INFO 2018/08/19 14:18:17 asynchttp.go:288: Started job 7665b435ac04671b9c6e840d2ba542db >[heketi] INFO 2018/08/19 14:18:17 Started async operation: Create Block Volume >[negroni] Started GET /queue/7665b435ac04671b9c6e840d2ba542db >[negroni] Completed 200 OK in 102.931µs >[cmdexec] INFO 2018/08/19 14:18:17 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 21min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:18:17 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:17 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:18:17 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 15min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:18:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_5ae7f1b350a435f5172dc67b7376b7e1 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_5ae7f1b350a435f5172dc67b7376b7e1[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_5ae7f1b350a435f5172dc67b7376b7e1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_5ae7f1b350a435f5172dc67b7376b7e1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:18:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/7665b435ac04671b9c6e840d2ba542db >[negroni] Completed 200 OK in 200.834µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:18:18 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 46.544648ms >[asynchttp] INFO 2018/08/19 14:18:18 asynchttp.go:288: Started job 573c2e9005cbc0fbeff8a7fe4dc43bde >[heketi] INFO 2018/08/19 14:18:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/573c2e9005cbc0fbeff8a7fe4dc43bde >[negroni] Completed 200 OK in 124.366µs >[cmdexec] INFO 2018/08/19 14:18:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_5ae7f1b350a435f5172dc67b7376b7e1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_5ae7f1b350a435f5172dc67b7376b7e1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:18 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:18:18 asynchttp.go:292: Completed job 7665b435ac04671b9c6e840d2ba542db in 1.264180023s >[kubeexec] DEBUG 2018/08/19 14:18:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:18:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/7665b435ac04671b9c6e840d2ba542db >[negroni] Completed 500 Internal Server Error in 165.841µs >[kubeexec] DEBUG 2018/08/19 14:18:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:18:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/573c2e9005cbc0fbeff8a7fe4dc43bde >[negroni] Completed 200 OK in 150.403µs >[kubeexec] DEBUG 2018/08/19 14:18:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 15min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:18:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/573c2e9005cbc0fbeff8a7fe4dc43bde >[negroni] Completed 200 OK in 158.182µs >[kubeexec] ERROR 2018/08/19 14:18:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c78a160aa1d5aa96b70548b7cee05c76 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_c78a160aa1d5aa96b70548b7cee05c76[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:18:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_c78a160aa1d5aa96b70548b7cee05c76 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_c78a160aa1d5aa96b70548b7cee05c76: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:18:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/573c2e9005cbc0fbeff8a7fe4dc43bde >[negroni] Completed 200 OK in 162.249µs >[kubeexec] ERROR 2018/08/19 14:18:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_c78a160aa1d5aa96b70548b7cee05c76 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:22 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_c78a160aa1d5aa96b70548b7cee05c76: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:22 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:18:22 asynchttp.go:292: Completed job 573c2e9005cbc0fbeff8a7fe4dc43bde in 3.322958534s >[heketi] ERROR 2018/08/19 14:18:22 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/573c2e9005cbc0fbeff8a7fe4dc43bde >[negroni] Completed 500 Internal Server Error in 173.932µs >[heketi] INFO 2018/08/19 14:18:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:18:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:18:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:18:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:18:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:18:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:18:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:18:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:18:47 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 40.921236ms >[asynchttp] INFO 2018/08/19 14:18:47 asynchttp.go:288: Started job b729fe44ef7ce2f2c20fc4799531a5d5 >[heketi] INFO 2018/08/19 14:18:47 Started async operation: Create Block Volume >[negroni] Started GET /queue/b729fe44ef7ce2f2c20fc4799531a5d5 >[negroni] Completed 200 OK in 96.793µs >[cmdexec] INFO 2018/08/19 14:18:47 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:18:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:18:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:18:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_987d12c37951b00000c0732d27a42773 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_987d12c37951b00000c0732d27a42773[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_987d12c37951b00000c0732d27a42773 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_987d12c37951b00000c0732d27a42773: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:48 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:18:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b729fe44ef7ce2f2c20fc4799531a5d5 >[negroni] Completed 200 OK in 105.655µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:18:49 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 50.955798ms >[asynchttp] INFO 2018/08/19 14:18:49 asynchttp.go:288: Started job e17806835af65feaed01f5fed5f268b9 >[heketi] INFO 2018/08/19 14:18:49 Started async operation: Create Block Volume >[negroni] Started GET /queue/e17806835af65feaed01f5fed5f268b9 >[negroni] Completed 200 OK in 105.37µs >[cmdexec] INFO 2018/08/19 14:18:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:18:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_987d12c37951b00000c0732d27a42773 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:49 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_987d12c37951b00000c0732d27a42773: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:49 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:18:49 asynchttp.go:292: Completed job b729fe44ef7ce2f2c20fc4799531a5d5 in 1.347967674s >[heketi] ERROR 2018/08/19 14:18:49 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:18:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:18:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/b729fe44ef7ce2f2c20fc4799531a5d5 >[negroni] Completed 500 Internal Server Error in 210.157µs >[kubeexec] DEBUG 2018/08/19 14:18:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:18:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e17806835af65feaed01f5fed5f268b9 >[negroni] Completed 200 OK in 145.524µs >[kubeexec] DEBUG 2018/08/19 14:18:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:18:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e17806835af65feaed01f5fed5f268b9 >[negroni] Completed 200 OK in 184.741µs >[kubeexec] ERROR 2018/08/19 14:18:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_93e9dfc40320db8427c036c6847c8f37 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_93e9dfc40320db8427c036c6847c8f37[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:18:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_93e9dfc40320db8427c036c6847c8f37 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:51 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_93e9dfc40320db8427c036c6847c8f37: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:18:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:18:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 22min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e17806835af65feaed01f5fed5f268b9 >[negroni] Completed 200 OK in 125.307µs >[kubeexec] ERROR 2018/08/19 14:18:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_93e9dfc40320db8427c036c6847c8f37 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:18:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_93e9dfc40320db8427c036c6847c8f37: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:52 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:18:52 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:18:52 asynchttp.go:292: Completed job e17806835af65feaed01f5fed5f268b9 in 3.360667217s >[negroni] Started GET /queue/e17806835af65feaed01f5fed5f268b9 >[negroni] Completed 500 Internal Server Error in 133.175µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:19:18 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.037455ms >[asynchttp] INFO 2018/08/19 14:19:18 asynchttp.go:288: Started job 0e55d6169dc2022bcb5474c53c91f6b7 >[heketi] INFO 2018/08/19 14:19:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/0e55d6169dc2022bcb5474c53c91f6b7 >[negroni] Completed 200 OK in 128.016µs >[cmdexec] INFO 2018/08/19 14:19:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:19:18 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 22min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:19:18 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:19:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:19:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_0d06b06c80778fb60837c579ad2d9b34 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_0d06b06c80778fb60837c579ad2d9b34[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_0d06b06c80778fb60837c579ad2d9b34 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_0d06b06c80778fb60837c579ad2d9b34: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:19:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/0e55d6169dc2022bcb5474c53c91f6b7 >[negroni] Completed 200 OK in 141.119µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:19:19 Free size 0 is less than the requested block volume size 1 >[kubeexec] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_0d06b06c80778fb60837c579ad2d9b34 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_0d06b06c80778fb60837c579ad2d9b34: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Completed 202 Accepted in 51.850162ms >[asynchttp] INFO 2018/08/19 14:19:19 asynchttp.go:288: Started job 6ba9c2cfc1c12cb9428f664298fb0605 >[heketi] INFO 2018/08/19 14:19:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/6ba9c2cfc1c12cb9428f664298fb0605 >[negroni] Completed 200 OK in 117.079µs >[cmdexec] INFO 2018/08/19 14:19:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[asynchttp] INFO 2018/08/19 14:19:19 asynchttp.go:292: Completed job 0e55d6169dc2022bcb5474c53c91f6b7 in 1.200182413s >[heketi] ERROR 2018/08/19 14:19:19 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:19:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 16min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:19:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 23min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:19:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/0e55d6169dc2022bcb5474c53c91f6b7 >[negroni] Completed 500 Internal Server Error in 144.162µs >[negroni] Started GET /queue/6ba9c2cfc1c12cb9428f664298fb0605 >[negroni] Completed 200 OK in 161.796µs >[kubeexec] DEBUG 2018/08/19 14:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:19:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:19:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f63964deb59d59ac2f065e70afdd34d1 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_f63964deb59d59ac2f065e70afdd34d1[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/6ba9c2cfc1c12cb9428f664298fb0605 >[negroni] Completed 200 OK in 157.05µs >[kubeexec] ERROR 2018/08/19 14:19:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f63964deb59d59ac2f065e70afdd34d1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f63964deb59d59ac2f065e70afdd34d1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:19:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6ba9c2cfc1c12cb9428f664298fb0605 >[negroni] Completed 200 OK in 209.046µs >[kubeexec] ERROR 2018/08/19 14:19:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f63964deb59d59ac2f065e70afdd34d1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:22 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f63964deb59d59ac2f065e70afdd34d1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:22 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:19:22 asynchttp.go:292: Completed job 6ba9c2cfc1c12cb9428f664298fb0605 in 3.172823857s >[heketi] ERROR 2018/08/19 14:19:22 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/6ba9c2cfc1c12cb9428f664298fb0605 >[negroni] Completed 500 Internal Server Error in 138.213µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:19:48 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.960849ms >[asynchttp] INFO 2018/08/19 14:19:48 asynchttp.go:288: Started job fa4635dc5af0bccfbe20809f64ef4918 >[heketi] INFO 2018/08/19 14:19:48 Started async operation: Create Block Volume >[negroni] Started GET /queue/fa4635dc5af0bccfbe20809f64ef4918 >[negroni] Completed 200 OK in 118.451µs >[cmdexec] INFO 2018/08/19 14:19:48 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 23min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:19:48 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 17min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:19:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:48 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:19:48 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_eeab53f916781f33479daad8b0cbb12a ha 3 auth enable prealloc full 10.70.46.152,10.70.47.54,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_eeab53f916781f33479daad8b0cbb12a[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_eeab53f916781f33479daad8b0cbb12a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_eeab53f916781f33479daad8b0cbb12a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:19:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fa4635dc5af0bccfbe20809f64ef4918 >[negroni] Completed 200 OK in 169.944µs >[kubeexec] DEBUG 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:19:49 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 82.569201ms >[asynchttp] INFO 2018/08/19 14:19:49 asynchttp.go:288: Started job ff82e870a456a41f3f292a95df686759 >[heketi] INFO 2018/08/19 14:19:49 Started async operation: Create Block Volume >[kubeexec] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_eeab53f916781f33479daad8b0cbb12a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_eeab53f916781f33479daad8b0cbb12a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/ff82e870a456a41f3f292a95df686759 >[negroni] Completed 200 OK in 209.698µs >[cmdexec] INFO 2018/08/19 14:19:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[asynchttp] INFO 2018/08/19 14:19:49 asynchttp.go:292: Completed job fa4635dc5af0bccfbe20809f64ef4918 in 1.250855203s >[heketi] ERROR 2018/08/19 14:19:49 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:19:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:19:50 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 23min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:19:50 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/fa4635dc5af0bccfbe20809f64ef4918 >[negroni] Completed 500 Internal Server Error in 156.361µs >[negroni] Started GET /queue/ff82e870a456a41f3f292a95df686759 >[negroni] Completed 200 OK in 138.129µs >[kubeexec] DEBUG 2018/08/19 14:19:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 17min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:19:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:19:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_e286da5d948917bfcfda9f1c40d6e723 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_e286da5d948917bfcfda9f1c40d6e723[No space left on device]" } >]: Stderr [] >[negroni] Started GET /queue/ff82e870a456a41f3f292a95df686759 >[negroni] Completed 200 OK in 217.914µs >[kubeexec] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e286da5d948917bfcfda9f1c40d6e723 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e286da5d948917bfcfda9f1c40d6e723: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:19:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:19:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 23min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ff82e870a456a41f3f292a95df686759 >[negroni] Completed 200 OK in 143.981µs >[kubeexec] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_e286da5d948917bfcfda9f1c40d6e723 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_e286da5d948917bfcfda9f1c40d6e723: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:19:52 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:19:53 asynchttp.go:292: Completed job ff82e870a456a41f3f292a95df686759 in 3.241425957s >[heketi] ERROR 2018/08/19 14:19:53 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/ff82e870a456a41f3f292a95df686759 >[negroni] Completed 500 Internal Server Error in 138.966µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:20:18 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 43.103921ms >[asynchttp] INFO 2018/08/19 14:20:18 asynchttp.go:288: Started job f195d2cdd49aab02b02cd60c6fdceab5 >[heketi] INFO 2018/08/19 14:20:18 Started async operation: Create Block Volume >[negroni] Started GET /queue/f195d2cdd49aab02b02cd60c6fdceab5 >[negroni] Completed 200 OK in 164.951µs >[cmdexec] INFO 2018/08/19 14:20:18 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:20:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:20:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 17min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:20:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_fce47721351fe9221b35e6fe817673e1 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_fce47721351fe9221b35e6fe817673e1[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_fce47721351fe9221b35e6fe817673e1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_fce47721351fe9221b35e6fe817673e1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:19 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:20:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f195d2cdd49aab02b02cd60c6fdceab5 >[negroni] Completed 200 OK in 153.311µs >[kubeexec] DEBUG 2018/08/19 14:20:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:20:20 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 59.32139ms >[asynchttp] INFO 2018/08/19 14:20:20 asynchttp.go:288: Started job ca0b8f7d4b038750fff3c99465c0ba92 >[heketi] INFO 2018/08/19 14:20:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/ca0b8f7d4b038750fff3c99465c0ba92 >[negroni] Completed 200 OK in 155.109µs >[cmdexec] INFO 2018/08/19 14:20:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:20:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_fce47721351fe9221b35e6fe817673e1 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:20 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_fce47721351fe9221b35e6fe817673e1: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:20 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:20:20 asynchttp.go:292: Completed job f195d2cdd49aab02b02cd60c6fdceab5 in 1.294386394s >[heketi] ERROR 2018/08/19 14:20:20 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:20:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:20:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f195d2cdd49aab02b02cd60c6fdceab5 >[negroni] Completed 500 Internal Server Error in 132.427µs >[kubeexec] DEBUG 2018/08/19 14:20:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:20:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ca0b8f7d4b038750fff3c99465c0ba92 >[negroni] Completed 200 OK in 128.307µs >[kubeexec] DEBUG 2018/08/19 14:20:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 17min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:20:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ca0b8f7d4b038750fff3c99465c0ba92 >[negroni] Completed 200 OK in 125.53µs >[kubeexec] ERROR 2018/08/19 14:20:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_210d1e40400ff639279146e6384bc287 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_210d1e40400ff639279146e6384bc287[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:20:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_210d1e40400ff639279146e6384bc287 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:22 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_210d1e40400ff639279146e6384bc287: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:22 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:20:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ca0b8f7d4b038750fff3c99465c0ba92 >[negroni] Completed 200 OK in 119.788µs >[kubeexec] ERROR 2018/08/19 14:20:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_210d1e40400ff639279146e6384bc287 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:23 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_210d1e40400ff639279146e6384bc287: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:23 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:23 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:20:23 asynchttp.go:292: Completed job ca0b8f7d4b038750fff3c99465c0ba92 in 3.349493706s >[negroni] Started GET /queue/ca0b8f7d4b038750fff3c99465c0ba92 >[negroni] Completed 500 Internal Server Error in 122.955µs >[heketi] INFO 2018/08/19 14:20:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:20:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:20:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:20:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:20:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:20:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:20:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:20:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:20:49 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 57.180379ms >[asynchttp] INFO 2018/08/19 14:20:49 asynchttp.go:288: Started job f1c2cf4670cad99789ac96389e1501ea >[heketi] INFO 2018/08/19 14:20:49 Started async operation: Create Block Volume >[negroni] Started GET /queue/f1c2cf4670cad99789ac96389e1501ea >[negroni] Completed 200 OK in 137.359µs >[cmdexec] INFO 2018/08/19 14:20:49 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:20:49 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:20:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:20:49 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:49 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_3feb03ba5e980bedd904a774de2751d8 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_3feb03ba5e980bedd904a774de2751d8[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_3feb03ba5e980bedd904a774de2751d8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_3feb03ba5e980bedd904a774de2751d8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:20:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f1c2cf4670cad99789ac96389e1501ea >[negroni] Completed 200 OK in 128.091µs >[kubeexec] DEBUG 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:20:50 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 47.243811ms >[asynchttp] INFO 2018/08/19 14:20:50 asynchttp.go:288: Started job e64a0503c2699858e36f5db705aefa59 >[heketi] INFO 2018/08/19 14:20:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/e64a0503c2699858e36f5db705aefa59 >[negroni] Completed 200 OK in 148.667µs >[cmdexec] INFO 2018/08/19 14:20:50 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_3feb03ba5e980bedd904a774de2751d8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_3feb03ba5e980bedd904a774de2751d8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:20:50 asynchttp.go:292: Completed job f1c2cf4670cad99789ac96389e1501ea in 1.305047135s >[heketi] ERROR 2018/08/19 14:20:50 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:20:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:20:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/f1c2cf4670cad99789ac96389e1501ea >[negroni] Completed 500 Internal Server Error in 176.239µs >[kubeexec] DEBUG 2018/08/19 14:20:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:20:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e64a0503c2699858e36f5db705aefa59 >[negroni] Completed 200 OK in 224.818µs >[kubeexec] DEBUG 2018/08/19 14:20:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 24min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:20:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e64a0503c2699858e36f5db705aefa59 >[negroni] Completed 200 OK in 143.669µs >[kubeexec] ERROR 2018/08/19 14:20:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_be75d1656c6e42b49e08d8c9e86740e7 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_be75d1656c6e42b49e08d8c9e86740e7[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_be75d1656c6e42b49e08d8c9e86740e7 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_be75d1656c6e42b49e08d8c9e86740e7: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:20:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:20:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 24min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e64a0503c2699858e36f5db705aefa59 >[negroni] Completed 200 OK in 132.496µs >[kubeexec] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_be75d1656c6e42b49e08d8c9e86740e7 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_be75d1656c6e42b49e08d8c9e86740e7: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:20:53 asynchttp.go:292: Completed job e64a0503c2699858e36f5db705aefa59 in 3.35650596s >[heketi] ERROR 2018/08/19 14:20:53 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/e64a0503c2699858e36f5db705aefa59 >[negroni] Completed 500 Internal Server Error in 124.644µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:21:19 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 52.751439ms >[asynchttp] INFO 2018/08/19 14:21:19 asynchttp.go:288: Started job 05b7344f6825c76a69126abc46fff6e9 >[heketi] INFO 2018/08/19 14:21:19 Started async operation: Create Block Volume >[negroni] Started GET /queue/05b7344f6825c76a69126abc46fff6e9 >[negroni] Completed 200 OK in 164.258µs >[cmdexec] INFO 2018/08/19 14:21:19 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:21:19 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:19 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 25min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:21:19 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:21:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_6a009d373454279a90e8d9662117ccc0 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_6a009d373454279a90e8d9662117ccc0[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_6a009d373454279a90e8d9662117ccc0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_6a009d373454279a90e8d9662117ccc0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:20 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:21:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/05b7344f6825c76a69126abc46fff6e9 >[negroni] Completed 200 OK in 123.797µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:21:20 Free size 0 is less than the requested block volume size 1 >[kubeexec] DEBUG 2018/08/19 14:21:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Completed 202 Accepted in 55.386833ms >[asynchttp] INFO 2018/08/19 14:21:20 asynchttp.go:288: Started job e6dfafdac37552a9427b7c9829bbc691 >[heketi] INFO 2018/08/19 14:21:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/e6dfafdac37552a9427b7c9829bbc691 >[negroni] Completed 200 OK in 142.008µs >[cmdexec] INFO 2018/08/19 14:21:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:21:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_6a009d373454279a90e8d9662117ccc0 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_6a009d373454279a90e8d9662117ccc0: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:21 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:21 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:21:21 asynchttp.go:292: Completed job 05b7344f6825c76a69126abc46fff6e9 in 1.342870871s >[kubeexec] DEBUG 2018/08/19 14:21:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 18min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:21:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/05b7344f6825c76a69126abc46fff6e9 >[negroni] Completed 500 Internal Server Error in 156.946µs >[kubeexec] DEBUG 2018/08/19 14:21:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:21:21 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/e6dfafdac37552a9427b7c9829bbc691 >[negroni] Completed 200 OK in 144.034µs >[kubeexec] DEBUG 2018/08/19 14:21:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 25min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:21:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e6dfafdac37552a9427b7c9829bbc691 >[negroni] Completed 200 OK in 134.594µs >[kubeexec] ERROR 2018/08/19 14:21:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_02703e4c487ed3a523c9f263cebca694 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_02703e4c487ed3a523c9f263cebca694[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:21:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_02703e4c487ed3a523c9f263cebca694 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:23 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_02703e4c487ed3a523c9f263cebca694: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:23 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:21:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/e6dfafdac37552a9427b7c9829bbc691 >[negroni] Completed 200 OK in 124.206µs >[kubeexec] ERROR 2018/08/19 14:21:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_02703e4c487ed3a523c9f263cebca694 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:24 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_02703e4c487ed3a523c9f263cebca694: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:24 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:24 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:21:24 asynchttp.go:292: Completed job e6dfafdac37552a9427b7c9829bbc691 in 3.479053479s >[negroni] Started GET /queue/e6dfafdac37552a9427b7c9829bbc691 >[negroni] Completed 500 Internal Server Error in 202.78µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:21:50 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 49.098051ms >[asynchttp] INFO 2018/08/19 14:21:50 asynchttp.go:288: Started job ab88c9cf6a79a44c47429f00fcb19460 >[heketi] INFO 2018/08/19 14:21:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/ab88c9cf6a79a44c47429f00fcb19460 >[negroni] Completed 200 OK in 100.003µs >[cmdexec] INFO 2018/08/19 14:21:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:21:50 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 19min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:21:50 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 25min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:21:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:21:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_43470a0fd1020c3778bf09b56648fac6 ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_43470a0fd1020c3778bf09b56648fac6[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_43470a0fd1020c3778bf09b56648fac6 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_43470a0fd1020c3778bf09b56648fac6: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:21:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ab88c9cf6a79a44c47429f00fcb19460 >[negroni] Completed 200 OK in 176.967µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:21:51 Free size 0 is less than the requested block volume size 1 >[kubeexec] DEBUG 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Completed 202 Accepted in 48.650716ms >[asynchttp] INFO 2018/08/19 14:21:51 asynchttp.go:288: Started job cd2a2cb41734d320174e7400e7a26102 >[heketi] INFO 2018/08/19 14:21:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/cd2a2cb41734d320174e7400e7a26102 >[negroni] Completed 200 OK in 102.632µs >[cmdexec] INFO 2018/08/19 14:21:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_43470a0fd1020c3778bf09b56648fac6 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_43470a0fd1020c3778bf09b56648fac6: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:51 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:21:51 asynchttp.go:292: Completed job ab88c9cf6a79a44c47429f00fcb19460 in 1.258323934s >[kubeexec] DEBUG 2018/08/19 14:21:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 19min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:21:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ab88c9cf6a79a44c47429f00fcb19460 >[negroni] Completed 500 Internal Server Error in 122.679µs >[kubeexec] DEBUG 2018/08/19 14:21:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 25min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:21:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/cd2a2cb41734d320174e7400e7a26102 >[negroni] Completed 200 OK in 124.05µs >[kubeexec] DEBUG 2018/08/19 14:21:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:21:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cd2a2cb41734d320174e7400e7a26102 >[negroni] Completed 200 OK in 179.068µs >[kubeexec] ERROR 2018/08/19 14:21:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_284b232fe1d3a6ac6596262c570a1cd8 ha 3 auth enable prealloc full 10.70.47.54,10.70.46.152,10.70.47.183 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_284b232fe1d3a6ac6596262c570a1cd8[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:21:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_284b232fe1d3a6ac6596262c570a1cd8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:53 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_284b232fe1d3a6ac6596262c570a1cd8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:53 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:21:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:21:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 25min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/cd2a2cb41734d320174e7400e7a26102 >[negroni] Completed 200 OK in 149.176µs >[kubeexec] ERROR 2018/08/19 14:21:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_284b232fe1d3a6ac6596262c570a1cd8 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:21:54 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_284b232fe1d3a6ac6596262c570a1cd8: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:21:54 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:21:54 asynchttp.go:292: Completed job cd2a2cb41734d320174e7400e7a26102 in 3.28137752s >[heketi] ERROR 2018/08/19 14:21:54 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/cd2a2cb41734d320174e7400e7a26102 >[negroni] Completed 500 Internal Server Error in 132.219µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:22:20 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 53.763613ms >[asynchttp] INFO 2018/08/19 14:22:20 asynchttp.go:288: Started job 6c42b0e33c37884726da3958856ff37f >[heketi] INFO 2018/08/19 14:22:20 Started async operation: Create Block Volume >[negroni] Started GET /queue/6c42b0e33c37884726da3958856ff37f >[negroni] Completed 200 OK in 149.604µs >[cmdexec] INFO 2018/08/19 14:22:20 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 19min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:22:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:22:20 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:20 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:22:20 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f56bf89c5b5f647b8f54cdc257a05e0d ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_f56bf89c5b5f647b8f54cdc257a05e0d[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f56bf89c5b5f647b8f54cdc257a05e0d --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f56bf89c5b5f647b8f54cdc257a05e0d: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:22:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6c42b0e33c37884726da3958856ff37f >[negroni] Completed 200 OK in 113.446µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:22:21 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 53.20934ms >[asynchttp] INFO 2018/08/19 14:22:21 asynchttp.go:288: Started job 3d7d7876bb9a68dc5f9da11127419785 >[heketi] INFO 2018/08/19 14:22:21 Started async operation: Create Block Volume >[kubeexec] DEBUG 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3d7d7876bb9a68dc5f9da11127419785 >[negroni] Completed 200 OK in 120.355µs >[cmdexec] INFO 2018/08/19 14:22:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f56bf89c5b5f647b8f54cdc257a05e0d --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f56bf89c5b5f647b8f54cdc257a05e0d: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:22:21 asynchttp.go:292: Completed job 6c42b0e33c37884726da3958856ff37f in 1.403140559s >[heketi] ERROR 2018/08/19 14:22:21 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[kubeexec] DEBUG 2018/08/19 14:22:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 19min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:22:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/6c42b0e33c37884726da3958856ff37f >[negroni] Completed 500 Internal Server Error in 183.802µs >[kubeexec] DEBUG 2018/08/19 14:22:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:22:22 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3d7d7876bb9a68dc5f9da11127419785 >[negroni] Completed 200 OK in 159.847µs >[kubeexec] DEBUG 2018/08/19 14:22:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:22:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3d7d7876bb9a68dc5f9da11127419785 >[negroni] Completed 200 OK in 126.363µs >[kubeexec] ERROR 2018/08/19 14:22:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_34d89b2ba730c15b15a95dea56f78695 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_34d89b2ba730c15b15a95dea56f78695[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_34d89b2ba730c15b15a95dea56f78695 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_34d89b2ba730c15b15a95dea56f78695: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:22:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3d7d7876bb9a68dc5f9da11127419785 >[negroni] Completed 200 OK in 149.799µs >[kubeexec] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_34d89b2ba730c15b15a95dea56f78695 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_34d89b2ba730c15b15a95dea56f78695: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:22:24 asynchttp.go:292: Completed job 3d7d7876bb9a68dc5f9da11127419785 in 3.276972385s >[heketi] ERROR 2018/08/19 14:22:24 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[negroni] Started GET /queue/3d7d7876bb9a68dc5f9da11127419785 >[negroni] Completed 500 Internal Server Error in 192.637µs >[heketi] INFO 2018/08/19 14:22:36 Starting Node Health Status refresh >[cmdexec] INFO 2018/08/19 14:22:36 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[heketi] INFO 2018/08/19 14:22:37 Periodic health check status: node 5f0be17866654828d51d78022320f1f8 up=true >[cmdexec] INFO 2018/08/19 14:22:37 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[heketi] INFO 2018/08/19 14:22:37 Periodic health check status: node 6120e5eb7b35b19b1389d33ad1cf9991 up=true >[cmdexec] INFO 2018/08/19 14:22:37 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:37 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[heketi] INFO 2018/08/19 14:22:37 Periodic health check status: node 9889f77e242b9e5b2edc1f07ea800b87 up=true >[heketi] INFO 2018/08/19 14:22:37 Cleaned 0 nodes from health cache >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:22:50 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 59.821993ms >[asynchttp] INFO 2018/08/19 14:22:50 asynchttp.go:288: Started job 3a5e53a7c5aa92582bede76d73cef9ba >[heketi] INFO 2018/08/19 14:22:50 Started async operation: Create Block Volume >[negroni] Started GET /queue/3a5e53a7c5aa92582bede76d73cef9ba >[negroni] Completed 200 OK in 106.733µs >[cmdexec] INFO 2018/08/19 14:22:50 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:50 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:22:50 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:22:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:22:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_f1475cf5f8037b30891e97d2c3b6b844 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_f1475cf5f8037b30891e97d2c3b6b844[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f1475cf5f8037b30891e97d2c3b6b844 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f1475cf5f8037b30891e97d2c3b6b844: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:51 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:22:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/3a5e53a7c5aa92582bede76d73cef9ba >[negroni] Completed 200 OK in 111.161µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:22:51 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 50.635348ms >[asynchttp] INFO 2018/08/19 14:22:51 asynchttp.go:288: Started job b6f614846bded7ca1d6bfa1e5305facd >[heketi] INFO 2018/08/19 14:22:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/b6f614846bded7ca1d6bfa1e5305facd >[negroni] Completed 200 OK in 187.789µs >[cmdexec] INFO 2018/08/19 14:22:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:22:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_f1475cf5f8037b30891e97d2c3b6b844 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_f1475cf5f8037b30891e97d2c3b6b844: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:52 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:52 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:22:52 asynchttp.go:292: Completed job 3a5e53a7c5aa92582bede76d73cef9ba in 1.229055888s >[kubeexec] DEBUG 2018/08/19 14:22:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:22:52 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 26min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:22:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/3a5e53a7c5aa92582bede76d73cef9ba >[negroni] Completed 500 Internal Server Error in 119.039µs >[negroni] Started GET /queue/b6f614846bded7ca1d6bfa1e5305facd >[negroni] Completed 200 OK in 207.047µs >[kubeexec] DEBUG 2018/08/19 14:22:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:22:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b6f614846bded7ca1d6bfa1e5305facd >[negroni] Completed 200 OK in 135.334µs >[kubeexec] ERROR 2018/08/19 14:22:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_6c671d0a73b69e335908979bc376e2a2 ha 3 auth enable prealloc full 10.70.47.183,10.70.46.152,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_6c671d0a73b69e335908979bc376e2a2[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:22:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_6c671d0a73b69e335908979bc376e2a2 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:54 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_6c671d0a73b69e335908979bc376e2a2: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:54 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:22:54 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:22:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 26min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/b6f614846bded7ca1d6bfa1e5305facd >[negroni] Completed 200 OK in 176.451µs >[kubeexec] ERROR 2018/08/19 14:22:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_6c671d0a73b69e335908979bc376e2a2 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:22:55 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_6c671d0a73b69e335908979bc376e2a2: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:55 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:22:55 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:22:55 asynchttp.go:292: Completed job b6f614846bded7ca1d6bfa1e5305facd in 3.362583481s >[negroni] Started GET /queue/b6f614846bded7ca1d6bfa1e5305facd >[negroni] Completed 500 Internal Server Error in 182.211µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:23:21 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 47.837488ms >[asynchttp] INFO 2018/08/19 14:23:21 asynchttp.go:288: Started job 5f275b0c9f732054389122653246d1a8 >[heketi] INFO 2018/08/19 14:23:21 Started async operation: Create Block Volume >[negroni] Started GET /queue/5f275b0c9f732054389122653246d1a8 >[negroni] Completed 200 OK in 114.233µs >[cmdexec] INFO 2018/08/19 14:23:21 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 27min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:23:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:23:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:23:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d58e65b32a56f071647ce1149f74ca15 ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_d58e65b32a56f071647ce1149f74ca15[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d58e65b32a56f071647ce1149f74ca15 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:21 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d58e65b32a56f071647ce1149f74ca15: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:21 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:23:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/5f275b0c9f732054389122653246d1a8 >[negroni] Completed 200 OK in 162.179µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:23:22 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.537951ms >[asynchttp] INFO 2018/08/19 14:23:22 asynchttp.go:288: Started job ca07ac68296fbde7653eac0d4158be9f >[heketi] INFO 2018/08/19 14:23:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/ca07ac68296fbde7653eac0d4158be9f >[negroni] Completed 200 OK in 118.271µs >[cmdexec] INFO 2018/08/19 14:23:22 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d58e65b32a56f071647ce1149f74ca15 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:22 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d58e65b32a56f071647ce1149f74ca15: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:22 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:22 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:23:22 asynchttp.go:292: Completed job 5f275b0c9f732054389122653246d1a8 in 1.290439471s >[kubeexec] DEBUG 2018/08/19 14:23:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 27min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:23:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/5f275b0c9f732054389122653246d1a8 >[negroni] Completed 500 Internal Server Error in 182.66µs >[kubeexec] DEBUG 2018/08/19 14:23:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:23:23 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[negroni] Started GET /queue/ca07ac68296fbde7653eac0d4158be9f >[negroni] Completed 200 OK in 187.627µs >[kubeexec] DEBUG 2018/08/19 14:23:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 20min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:23:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ca07ac68296fbde7653eac0d4158be9f >[negroni] Completed 200 OK in 141.9µs >[kubeexec] ERROR 2018/08/19 14:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_c6944a0285e8719f18459d47e60da64f ha 3 auth enable prealloc full 10.70.46.152,10.70.47.183,10.70.47.54 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_c6944a0285e8719f18459d47e60da64f[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:23:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_c6944a0285e8719f18459d47e60da64f --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:24 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_c6944a0285e8719f18459d47e60da64f: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:24 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:23:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/ca07ac68296fbde7653eac0d4158be9f >[negroni] Completed 200 OK in 118.935µs >[kubeexec] ERROR 2018/08/19 14:23:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_c6944a0285e8719f18459d47e60da64f --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:25 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_c6944a0285e8719f18459d47e60da64f: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:25 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:25 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:23:25 asynchttp.go:292: Completed job ca07ac68296fbde7653eac0d4158be9f in 3.316345465s >[negroni] Started GET /queue/ca07ac68296fbde7653eac0d4158be9f >[negroni] Completed 500 Internal Server Error in 134.045µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:23:51 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 52.081786ms >[asynchttp] INFO 2018/08/19 14:23:51 asynchttp.go:288: Started job 23aea14fc418b54a47d22d48f1e68f22 >[heketi] INFO 2018/08/19 14:23:51 Started async operation: Create Block Volume >[negroni] Started GET /queue/23aea14fc418b54a47d22d48f1e68f22 >[negroni] Completed 200 OK in 104.274µs >[cmdexec] INFO 2018/08/19 14:23:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:23:51 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 21min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:23:51 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:51 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 27min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:23:51 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_98c72c68bb876bef19cedbfe9bf0e0af ha 3 auth enable prealloc full 10.70.47.183,10.70.47.54,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_98c72c68bb876bef19cedbfe9bf0e0af[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_98c72c68bb876bef19cedbfe9bf0e0af --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_98c72c68bb876bef19cedbfe9bf0e0af: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:23:52 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/23aea14fc418b54a47d22d48f1e68f22 >[negroni] Completed 200 OK in 96.055µs >[kubeexec] DEBUG 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:23:52 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 48.539044ms >[asynchttp] INFO 2018/08/19 14:23:52 asynchttp.go:288: Started job 2e3936f92a0c8d9b8a3e7893ce181701 >[heketi] INFO 2018/08/19 14:23:52 Started async operation: Create Block Volume >[negroni] Started GET /queue/2e3936f92a0c8d9b8a3e7893ce181701 >[negroni] Completed 200 OK in 121.187µs >[cmdexec] INFO 2018/08/19 14:23:52 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_98c72c68bb876bef19cedbfe9bf0e0af --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_98c72c68bb876bef19cedbfe9bf0e0af: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:52 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:23:52 asynchttp.go:292: Completed job 23aea14fc418b54a47d22d48f1e68f22 in 1.333624506s >[kubeexec] DEBUG 2018/08/19 14:23:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 21min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:23:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/23aea14fc418b54a47d22d48f1e68f22 >[negroni] Completed 500 Internal Server Error in 122.534µs >[kubeexec] DEBUG 2018/08/19 14:23:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:23:53 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/2e3936f92a0c8d9b8a3e7893ce181701 >[negroni] Completed 200 OK in 149.497µs >[kubeexec] DEBUG 2018/08/19 14:23:53 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 27min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:23:53 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2e3936f92a0c8d9b8a3e7893ce181701 >[negroni] Completed 200 OK in 120.685µs >[kubeexec] ERROR 2018/08/19 14:23:54 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_d0bda42435975a2238068e00ae380f23 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_d0bda42435975a2238068e00ae380f23[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d0bda42435975a2238068e00ae380f23 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d0bda42435975a2238068e00ae380f23: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:23:55 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:23:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 27min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/2e3936f92a0c8d9b8a3e7893ce181701 >[negroni] Completed 200 OK in 122.39µs >[kubeexec] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_d0bda42435975a2238068e00ae380f23 --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_d0bda42435975a2238068e00ae380f23: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:55 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:23:56 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:23:56 asynchttp.go:292: Completed job 2e3936f92a0c8d9b8a3e7893ce181701 in 3.397735748s >[negroni] Started GET /queue/2e3936f92a0c8d9b8a3e7893ce181701 >[negroni] Completed 500 Internal Server Error in 195.356µs >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:24:21 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.345832ms >[asynchttp] INFO 2018/08/19 14:24:21 asynchttp.go:288: Started job 4df3fba207ac13d4a724a1ff3693c005 >[heketi] INFO 2018/08/19 14:24:21 Started async operation: Create Block Volume >[negroni] Started GET /queue/4df3fba207ac13d4a724a1ff3693c005 >[negroni] Completed 200 OK in 140.277µs >[cmdexec] INFO 2018/08/19 14:24:21 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:21 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 21min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:24:21 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:24:22 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 28min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:24:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[kubeexec] ERROR 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_eee6258568af5cc1ee8257556f23bb7a ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_eee6258568af5cc1ee8257556f23bb7a[No space left on device]" } >]: Stderr [] >[kubeexec] ERROR 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_eee6258568af5cc1ee8257556f23bb7a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_eee6258568af5cc1ee8257556f23bb7a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:24:22 /src/github.com/heketi/heketi/apps/glusterfs/operations.go:828: Error executing create block volume: Unable to execute command on glusterfs-storage-nr58s: >[cmdexec] INFO 2018/08/19 14:24:22 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4df3fba207ac13d4a724a1ff3693c005 >[negroni] Completed 200 OK in 125.028µs >[kubeexec] DEBUG 2018/08/19 14:24:22 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started POST /blockvolumes >[heketi] WARNING 2018/08/19 14:24:22 Free size 0 is less than the requested block volume size 1 >[negroni] Completed 202 Accepted in 45.322774ms >[asynchttp] INFO 2018/08/19 14:24:22 asynchttp.go:288: Started job 6cddcc14949879a6a06305c7992d288c >[heketi] INFO 2018/08/19 14:24:22 Started async operation: Create Block Volume >[negroni] Started GET /queue/6cddcc14949879a6a06305c7992d288c >[negroni] Completed 200 OK in 154.197µs >[cmdexec] INFO 2018/08/19 14:24:22 Check Glusterd service status in node dhcp47-54.lab.eng.blr.redhat.com >[kubeexec] ERROR 2018/08/19 14:24:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block delete vol_3589da219d6536edb00cf7b533976e25/blockvol_eee6258568af5cc1ee8257556f23bb7a --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL" } >]: Stderr [] >[cmdexec] ERROR 2018/08/19 14:24:23 /src/github.com/heketi/heketi/executors/cmdexec/block_volume.go:102: Unable to delete volume blockvol_eee6258568af5cc1ee8257556f23bb7a: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:24:23 /src/github.com/heketi/heketi/apps/glusterfs/block_volume_entry.go:315: Unable to delete volume: Unable to execute command on glusterfs-storage-nr58s: >[heketi] ERROR 2018/08/19 14:24:23 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:113: Create Block Volume Failed: Unable to execute command on glusterfs-storage-nr58s: >[asynchttp] INFO 2018/08/19 14:24:23 asynchttp.go:292: Completed job 4df3fba207ac13d4a724a1ff3693c005 in 1.217925378s >[kubeexec] DEBUG 2018/08/19 14:24:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-54.lab.eng.blr.redhat.com Pod: glusterfs-storage-pf27l Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 13:02:33 UTC; 1h 21min ago > Process: 238 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 240 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0467d513_a3b0_11e8_9369_005056a51f55.slice/docker-793021c83126a039820109e3bf2e3c12b29976d4f35a8276e9aff7aaa2b20d16.scope/system.slice/glusterd.service > ââ240 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ455 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/5354b44c224ed2c52d98bfffd91b4407.socket --xlator-option *replicate*.node-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d > ââ480 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id blkhosting.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick -p /var/run/gluster/vols/blkhosting/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.pid -S /var/run/gluster/753737c3a98d855a960084e537ff8d54.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_76fe877734965f6306b87a14b9e73293/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_76fe877734965f6306b87a14b9e73293-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ489 /usr/sbin/glusterfsd -s 10.70.47.54 --volfile-id heketidbstorage.10.70.47.54.var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.54-var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.pid -S /var/run/gluster/f68a7b751cdeb40d644979a12160a720.socket --brick-name /var/lib/heketi/mounts/vg_3fecdb9fc84903807eef185ca1057930/brick_d214f841f456423a147bb31fb225fd78/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_3fecdb9fc84903807eef185ca1057930-brick_d214f841f456423a147bb31fb225fd78-brick.log --xlator-option *-posix.glusterd-uuid=ee49d270-8512-49a0-b4a6-601b3a403f5d --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 >[cmdexec] INFO 2018/08/19 14:24:23 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[cmdexec] INFO 2018/08/19 14:24:23 Check Glusterd service status in node dhcp46-152.lab.eng.blr.redhat.com >[negroni] Started GET /queue/4df3fba207ac13d4a724a1ff3693c005 >[negroni] Completed 500 Internal Server Error in 191.188µs >[negroni] Started GET /queue/6cddcc14949879a6a06305c7992d288c >[negroni] Completed 200 OK in 235.4µs >[kubeexec] DEBUG 2018/08/19 14:24:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp46-152.lab.eng.blr.redhat.com Pod: glusterfs-storage-vsh6t Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:18 UTC; 2h 28min ago > Process: 214 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 216 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5571c09_a176_11e8_9e7f_005056a51f55.slice/docker-18dc45ad2f44574a46ccf04df3b5c53498c28724a1dc4a44fb64f9703644b646.scope/system.slice/glusterd.service > ââ 216 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 495 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id blkhosting.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick -p /var/run/gluster/vols/blkhosting/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.pid -S /var/run/gluster/3c77facc5e8d6cf869f0d9741f4c172a.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_82bac3785b8184caa888c313acbf5815/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_82bac3785b8184caa888c313acbf5815-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49152 --xlator-option blkhosting-server.listen-port=49152 > ââ 521 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id heketidbstorage.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick -p /var/run/gluster/vols/heketidbstorage/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.pid -S /var/run/gluster/89e7191ff4070aeee75c43b2d03a55f4.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_a2837129de8dc24c6590ce680c7453a2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_a2837129de8dc24c6590ce680c7453a2-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 4795 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_80ff41fdbb475c0fa055de08db90ab4f.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick -p /var/run/gluster/vols/vol_80ff41fdbb475c0fa055de08db90ab4f/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.pid -S /var/run/gluster/1c2323f54eb45aff23ad1b85a5016b88.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_86bf1fe4ee504b97331bdf87ded5f601/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_86bf1fe4ee504b97331bdf87ded5f601-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49154 --xlator-option vol_80ff41fdbb475c0fa055de08db90ab4f-server.listen-port=49154 > ââ 6161 /usr/sbin/glusterfsd -s 10.70.46.152 --volfile-id vol_45a4ba9ee3f5a61061b35d95ae58adc8.10.70.46.152.var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick -p /var/run/gluster/vols/vol_45a4ba9ee3f5a61061b35d95ae58adc8/10.70.46.152-var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.pid -S /var/run/gluster/558f9c0e09d4d5f7ed04b9e1e027773d.socket --brick-name /var/lib/heketi/mounts/vg_af2c81ded89e5ffa84b759f5fc717bc7/brick_ef9430b4e5227d717f3321947cb741c9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_af2c81ded89e5ffa84b759f5fc717bc7-brick_ef9430b4e5227d717f3321947cb741c9-brick.log --xlator-option *-posix.glusterd-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 --brick-port 49155 --xlator-option vol_45a4ba9ee3f5a61061b35d95ae58adc8-server.listen-port=49155 > ââ10104 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/d20abc74d8025a94ddb90d11e4c58a05.socket --xlator-option *replicate*.node-uuid=c501c122-9937-4b4c-95cc-12e5fb3d3975 >[cmdexec] INFO 2018/08/19 14:24:24 Check Glusterd service status in node dhcp47-183.lab.eng.blr.redhat.com >[kubeexec] DEBUG 2018/08/19 14:24:24 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: dhcp47-183.lab.eng.blr.redhat.com Pod: glusterfs-storage-nr58s Command: systemctl status glusterd >Result: â glusterd.service - GlusterFS, a clustered file-system server > Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) > Active: active (running) since Sun 2018-08-19 11:56:10 UTC; 2h 28min ago > Process: 205 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) > Main PID: 207 (glusterd) > CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda555347f_a176_11e8_9e7f_005056a51f55.slice/docker-6446c9a5ef0df86308a3e53e25e07e56311cab816ee62c7428881b323f9cb950.scope/system.slice/glusterd.service > ââ 207 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO > ââ 694 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 > ââ 706 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id blkhosting.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick -p /var/run/gluster/vols/blkhosting/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.pid -S /var/run/gluster/852236e5ad03a083938266561d53d9d8.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_fc228e7435b4881233ffe0fd3c222c77/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_fc228e7435b4881233ffe0fd3c222c77-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49154 --xlator-option blkhosting-server.listen-port=49154 > ââ 712 /usr/sbin/glusterfsd -s 10.70.47.183 --volfile-id heketidbstorage.10.70.47.183.var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick -p /var/run/gluster/vols/heketidbstorage/10.70.47.183-var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.pid -S /var/run/gluster/5ad64fd5618d695a3c7e209b6a93a8a7.socket --brick-name /var/lib/heketi/mounts/vg_f89b9b3b7340e500f2c6367273182b28/brick_f4cc9aa56c8efb4c60285d771bd5fca9/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_f89b9b3b7340e500f2c6367273182b28-brick_f4cc9aa56c8efb4c60285d771bd5fca9-brick.log --xlator-option *-posix.glusterd-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 --brick-port 49155 --xlator-option heketidbstorage-server.listen-port=49155 > ââ11652 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/150b377103f5f726f777a10d43c66a88.socket --xlator-option *replicate*.node-uuid=27d4591d-5f1e-4861-b5e4-15f5da964895 >[negroni] Started GET /queue/6cddcc14949879a6a06305c7992d288c >[negroni] Completed 200 OK in 200.469µs >[kubeexec] ERROR 2018/08/19 14:24:25 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:242: Failed to run command [gluster-block create vol_3589da219d6536edb00cf7b533976e25/blockvol_417a2d1fe937f7fdc54acd190bd696a2 ha 3 auth enable prealloc full 10.70.47.54,10.70.47.183,10.70.46.152 1GiB --json] on glusterfs-storage-nr58s: Err[command terminated with exit code 255]: Stdout [{ "RESULT": "FAIL", "errCode": 255, "errMsg": "Failed to update transaction log for vol_3589da219d6536edb00cf7b533976e25\/blockvol_417a2d1fe937f7fdc54acd190bd696a2[No space left on device]" } >]: Stderr []
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1476922">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1619017
: 1476922 |
1476923
|
1477857